Control and/or registration of smart devices, locally by an assistant client device

ABSTRACT

Various implementations relate to generating, locally at an assistant client device, specific control commands that, when transmitted to a corresponding smart device, are directly interpretable by the corresponding smart device to effectuate a state change at the corresponding smart device, or at a corresponding additional smart device directly controlled by the corresponding smart device. Various implementations additionally or alternatively relate to utilizing local assistant client devices in discovering, provisioning, and/or registering smart devices for an account of a user.

BACKGROUND

Humans can engage in human-to-computer interactions with interactivesoftware applications referred to herein as “automated assistants”. Forexample, a human (which when interacting with an automated assistant maybe referred to as a “user”) may provide an input to the automatedassistant that can cause the automated assistant to generate and provideresponsive output, to control one or more smart devices, and/or toperform one or more other functionalities. The input provided by theuser can be, for example, a touch input (e.g., via a touchscreen), agesture (e.g., detected via a camera), and/or a spoken natural languageinput (i.e., utterance detected via microphone(s)), which may in somecases be converted into text (or other semantic representation) and thenfurther processed.

In many cases, automated assistants include automated assistant clientsthat are executed locally by assistant client devices and that areengaged with directly by users, as well as cloud-based counterpart(s)that leverage the more robust resources of the cloud to help automatedassistant clients respond to users’ inputs. For example, an automatedassistant client can provide, to the cloud-based counterpart(s), anaudio recording of a spoken utterance of a user (or a text conversionthereof), and optionally data indicative of the user’s identity (e.g.,credentials). The cloud-based counterpart may perform various processingon the query to return result(s) to the automated assistant client,which may then provide corresponding output to the user.

Many users may engage automated assistants using multiple clientdevices. For example, some users may possess a coordinated “ecosystem”of client devices such as one or more smart phones, one or more tabletcomputers, one or more vehicle computing systems, one or wearablecomputing devices, one or more smart televisions, one or more standaloneassistant-centric interactive speakers, one or more standaloneassistant-centric interactive displays with speaker(s), among otherclient devices. A user may engage in human-to-computer dialog with anautomated assistant using any of these client devices (assuming anautomated assistant client is installed). In some cases these clientdevices may be scattered around the user’s primary residence, secondaryresidence, workplace, and/or other structure(s). For example, mobileclient devices such as smart phones, tablets, smart watches, etc., maybe on the user’s person and/or wherever the user last placed them. Otherclient devices, such as traditional desktop computers, smarttelevisions, and standalone assistant-centric devices may be morestationary, but nonetheless may be located at various places (e.g.,rooms) within the user’s home or workplace.

Techniques exist to enable user(s) (e.g., a single user, multiple usersin a family, coworkers, co-inhabitants, etc.) to utilize an automatedassistant client of any one of a coordinated ecosystem of client devicesto control any one of a plurality of smart devices that are associatedwith the automated assistant client. For example, a user can issue aspoken command of “turn off the living room lights” to an automatedassistant client of a client device to cause corresponding smart devices(i.e., lights linked to the automated assistant client and labeled as“living room” lights) to be turned off.

In controlling a smart device responsive to user input received at aclient device, many existing techniques transmit, via the Internet, datacorresponding to the user input, to remote automated assistantserver(s). The remote automated assistant server(s) process the data todetermine a smart device to be controlled based on the request, andtransmit, via the Internet, a request to server(s) of a separate partyassociated with the smart device (e.g., a manufacturer of the smartdevice). The server(s) of the separate party receive the request, thentransmit, via the Internet, corresponding command(s) to the smartdevice, whether through a hub co-present with the smart device (e.g., inthe case of BLE, Z-Wave, ZigBee, etc.) or to the smart device directlyvia an IP connection (e.g., in the case of Wi-Fi and other smart deviceswhich don’t require a hub). However, such techniques present drawbackssuch as high latency, low reliability, and/or excessive consumption ofnetwork resources. For example, high latency can be a result of thetransmission of the request from the remote assistant server(s) to theseparate party server(s), which is often exacerbated by the remoteassistant server(s) and separate party server(s) not beinggeographically proximate to one another. Also, for example, lowreliability can be due to the separate party server(s) failing torespond due to, for instance, server outages, network losses, etc.Further, such techniques can burden computational and/or networkresources.

Other approaches for control of smart devices can also presentdrawbacks. For example, some techniques can utilize a central hub thatutilizes only one or more standard protocols. However, such standardprotocols can fail for unspecified corners of the standard protocols,can fail for smart devices that fail to fully conform to a standardprotocol, fail for smart devices that fail to adhere to any standardprotocol, and/or can inhibit finer grained control of smart devices bysmart device manufacturers, thereby leading to suboptimal performance ofthe smart devices. Further, firmware updates of the smart devices canbreak integration with the central hub. Yet further, failure of thecentral hub and/or poor signals quality from placement of the centralhub can lead to inability of the central hub to locally control varioussmart devices.

SUMMARY

Various implementations described herein relate to generating, locallyat an assistant client device, specific control commands that, whentransmitted to a corresponding smart device, are directly interpretableby the corresponding smart device to effectuate a state change at thecorresponding smart device, or at a corresponding additional smartdevice directly controlled by the corresponding smart device. In some ofthose various implementations, the assistant client device locallystores a plurality of third-party (3P) adapters and/or a first-party(1P) adapter. A third-party or 3P, as used herein, references a partythat is distinct from the party that controls an automated assistantbeing referenced. A first-party or 1P, as used herein, references theparty that controls the automated assistant being referenced.

Each of the adapters can be provided by a corresponding party and, whenexecuted by the assistant client device, can process generic smartdevice control commands to generate specific control commands that areeach tailored, when locally transmitted to at least one correspondingsmart device of the party, to be directly interpretable by thecorresponding smart device to effectuate a state change at thecorresponding smart device, or at a corresponding additional smartdevice directly controlled by the corresponding smart device (e.g., inthe situations where the specific control commands are transmitted to ahub/bridge of the party, that then controls one or more additional smartdevices based on the specific control commands). For example, a 3Padapter can be implemented by 3P provided JavaScript (or otherinterpreted programming language) and can translate generic smart devicecontrol commands into specific control commands that conform to aprotocol suite of the 3P. Generic smart device control commands, as usedherein, can reference commands that convey smart device(s) and state(s)to be altered at the smart device(s), but that, if transmitted to thesmart device(s) would not cause the state(s) to be altered. The genericsmart device control commands can conform to a schema of the automatedassistant. For example, the generic smart device control command can bea structured command that defines an intent (e.g., turn on/off, dim,increase, decrease), defines a smart device (e.g., a unique identifierof the smart device), defines a third-party for the smart device (e.g.,a unique identifier for the third-party), and/or one or more furtherparameters for the intent (e.g., how much to dim, what toincrease/decrease, etc.). The 3P adapter can execute in a containerwithin the automated assistant client, can process generic smart devicecontrol commands as described, and can further optionally report eventsto the automated assistant client using the schema of the automatedassistant (i.e., translating from 3P specific event formats to theautomated assistant schema).

Each of the adapters can be assigned, locally at the assistant clientdevice, to at least one corresponding communication channel, of two ormore communication channels available at the assistant client device.For example, a first 3P adapter can be assigned to a Bluetooth radiochannel, of the available communication channels, and, as a result, theBluetooth radio channel can be utilized for transmitting specificcontrol commands generated by the first 3P adapter. Also, for example, asecond 3P adapter can be assigned to a Wi-Fi radio channel, of theavailable communication channels, and, as a result, the Wi-Fi radiochannel can be utilized for transmitting specific control commandsgenerated by the second 3P adapter. Also, for example, a third 3Padapter can be assigned to a low-rate wireless personal area networkradio channel, of the available communication channels, and, as aresult, that channel can be utilized for transmitting specific controlcommands generated by the second 3P adapter. As yet another example, afourth 3P adapter can be assigned to both a Bluetooth radio channel anda Wi-Fi radio channel, and, as a result specific control commands can betransmitted over either (or both) of the channels. Optionally, the samespecific control command can be transmitted over either channel toeffectuate the same control of a given 3P smart device (that can receivecontrol commands via either channel). While each communication channelcan utilize a different radio (or a different subset of radios), it isnoted that a given radio can optionally be utilized to transmit via anyone of multiple available protocol suites. For example, a Wi-Fi radiochannel can be utilized to transmit data using TCP, using UDP, or usingvariations thereof (e.g. variations in one or more layers of TCP or UDP,such as an application layer).

While each 3P adapter is assigned to a corresponding communicationchannel of the client device and that corresponding communicationchannel is utilized to transmit specific control commands generated bythe 3P adapter, the 3P adapter can optionally be prevented from directlyaccessing the corresponding communication channel. Rather, access to thecorresponding communication channel can be controlled by the automatedassistant client, and the automated assistant client can optionallyperform one or more verifications of specific control commands beforetransmitting the specific control command via the correspondingcommunication channel. For example, transmission of a specific controlcommand via a corresponding communication channel can be contingent onverification, by the automated assistant client, that the specificcontrol command is addressed to only the smart device(s) identified in acorresponding generic smart device control command for which thespecific control command was generated. Such verification can preventintentional or unintentional control of smart device(s) that are notspecified by the generic smart device control command. Additionally oralternatively, such verification can prevent other types of accidentalor malicious behavior such as flooding a Bluetooth mesh network withmessages to render it inoperable.

Each of the adapters can utilize interpreted programming language toconvert generic smart device control commands to corresponding specificcontrol commands that are specific to the adapter. The specific controlcommands generated by a 3P adapter conform to a protocol suite of the 3Padapter, thereby enabling each of the 3Ps to utilize a standard protocolsuite, a variation of the standard protocol suite, and/or their ownproprietary protocol suite. For example, a first 3P adapter can generateone or more specific control commands that fully conform to a firstprotocol suite that is defined as an industry standard. A second 3Padapter can generate one or more specific control commands that includeone or more variations to the first protocol suite. For instance, one ormore layers of the protocol suite can be adapted by the second 3P suchthat they differ from, and therefore do not conform with, thecorresponding layer(s) of the first protocol suite defined as anindustry standard. A third 3P adapter can generate one or more specificcontrol commands that are proprietary to the corresponding 3P, and thatdo not fully conform to any industry standard protocol suite.

Accordingly, 3P adapters described herein can be tailored, by acorresponding 3P, to a corresponding protocol suite. This enables theassistant client device to transmit corresponding specific controlcommands, even when they are for unspecified corners of standardprotocols, fail to fully conform to a standard protocol, and/or fail toadhere to any standard protocol. Thus, the assistant client device cansupport local control of a greater quantity of smart devices. Moreover,this enables a 3P to implement a 3P adapter that enables control ofcorresponding smart device(s), where the control is finer grained thanthat which would be available utilizing only a standard protocol. Thiscan lead to more optimal local control of the corresponding smartdevice(s). For instance, this can lead to rendering of light output,from a smart light, that is in greater conformance (e.g., in terms ofcolor or brightness) with a user request, than that available had astandard protocol instead been utilized. Further, the 3P adapters can beupdated periodically, on-demand, and/or at other intervals, therebyenabling the corresponding 3Ps to implement 3P adapters that conform tofirmware updates of their corresponding smart devices, and preventingerrors that would otherwise result if the firmware updates and standardprotocols utilized by a central hub were non-compatible. Yet further,local control of various smart devices can be accomplished via any oneof a plurality of assistant client devices in a coordinated ecosystem,thereby mitigating connectivity concerns, power failure concerns, orother failure concerns when only a single central smart hub is reliedupon to control multiple smart devices.

In some implementations, 3P adapter(s) are downloaded to an assistantclient device, from a remote server, responsive to smart device(s) ofthe 3P(s) being registered for the client device. For example, smartdevice(s) of a 3P can be registered, locally and/or non-locally, for auser account associated with each of a plurality of assistant clientdevices and, in response (and optionally during the registrationprocedure and optionally before full registration), a 3P adapter for the3P can be pushed to those assistant client devices for local storage ofthe 3P adapter at those assistant client devices for execution by thoseclient devices. Having the 3P adapter locally stored at the assistantclient devices can result in quick execution of the 3P adapter at theclient devices (and even preemptive execution as described herein).Moreover, by downloading the 3P adapter responsive to a corresponding 3Psmart device being registered (e.g., optionally during the registrationprocedure), the client devices can store only a subset of available 3Padapters (e.g., only those corresponding to smart devices registered toaccount(s) of the client device(s)), thereby conserving the oftenlimited local storage space available at assistant client devices by notlocally storing one or more (e.g., any) 3P adapters for non-registeredsmart devices.

Further, in some implementations a 3P adapter can be downloaded to onlya subset (e.g., to only one) of a plurality of assistant client devicesassociated with a user account. In some of those implementations, the 3Padapter is downloaded to those assistant device(s) having the strongestsignal strength(s), for a corresponding communication channel, withcorresponding 3P smart device(s). For example, a particular 3P adapterthat is assigned to a Bluetooth communication channel can be downloadedat a given assistant client device associated with a user account, basedon a signal strength for the Bluetooth communication channel, at thegiven assistant client device and between the given assistant clientdevice and smart device(s) of the 3P, satisfying one or more criteria.For instance, the one or more criteria can include that the signalstrength satisfies a threshold, such as being the greatest signalstrength, amongst all assistant client device associated with the useraccount. In such an instance, the particular 3P adapter can optionallybe downloaded to the given assistant client device without it beingdownloaded to any other of the assistant client devices associated withthe user account. In these and other manners, different assistant clientdevices associated with the user account can download different subsetsof 3P adapters based on signal strengths and/or other criteria. This canenable the assistant client devices associated with a user account tocollectively support a wide variety of 3P adapters, withoutnecessitating storage of each 3P adapter at each of the assistant clientdevices. This can be particularly beneficial in view of the oftenlimited local storage space available at assistant client devices. Forexample, absent such techniques a given assistant client device may nothave sufficient local storage to store all 3P adapters needed for a useraccount. However, utilizing such techniques can enable the needed 3Padapters to be intelligently distributed amongst the storage of multipleassistant client devices associated with the user account. In someimplementations, data indicating signal strength(s) (if any) each of theassistant client devices associated with a user account can be provided,by those devices, to a remote server or to one or more of the assistantclient devices. The server and/or the assistant client device(s) canutilize the data, and one or more criteria (e.g., criteria describedabove) in determining which assistant client devices should receivewhich 3P adapters. Corresponding 3P adapters can then be pushed to thecorresponding client devices, or data transmitted to the correspondingclient devices to cause the corresponding client devices to request orretrieve the corresponding 3P adapters.

Moreover, in some of the implementations where 3P adapter(s) aredownloaded to an assistant client device, the assistant client devicecan optionally preemptively execute one or more (e.g., all) of thelocally stored 3P adapters. Preemptively executing a 3P adapter, as usedherein, references executing the 3P adapter prior to a generic smartdevice control command, that is directed to smart device(s) of thecorresponding 3P and processed using the executing 3P adapter, beinggenerated. Preemptively executing a 3P adapter on an assistant clientdevice can include at least loading the 3P adapter into in-memory cacheof the assistant client device. Pre-emptively executing the 3P adaptercan reduce latency (relative to not preemptively executing the 3Padapter) in generating a specific control command using the 3P adapterand, resultantly, in transmitting the specific control command andeffectuating a corresponding state change. Thus, state changes at smartdevices can be effectuated with reduced latency.

In some implementations, all 3P adapters locally downloaded to anassistant client device are preemptively executed by the client device.In some other implementations, only a subset of the available 3Padapters are preemptively executed at a given time in view of memoryconstraints, processor constraints, and/or other considerations. In someof those other implementations, the subset that is preemptively executedat the given time can be based on one or more dynamic criteria. The oneor more dynamic criteria can include: a corresponding recency ofutilization of the particular 3P adapter at the client device; a signalstrength, between the client device and the particular smart devicecorresponding to the particular 3P adapter, for a communication channel;a time of day; and/or a day of the week. For example, the N mostrecently utilized (i.e., utilized in generating specific controlcommands) 3P adapters can be executed at a given assistant clientdevice. Also, for example, a particular 3P adapter can be executed at agiven assistant client device during one or more temporal periods (e.g.,time(s) of day and/or day(s) of the week) based on one or more pastoccurrences of utilization of the particular 3P adapter during or nearthose temporal period(s). As yet another example, a particular 3Padapter that is assigned to a Bluetooth communication channel can bepreemptively executed at a given assistant client device based on asignal strength for the Bluetooth communication channel, at the givenassistant client device and between the assistant client device andsmart device(s) of the 3P, satisfying a threshold (e.g., a fixedthreshold, or a threshold relative to other assistant client device(s)).It will be appreciated from the preceding and/or other descriptionherein, that at any given time different assistant client devices in anecosystem can have one or more different preemptively executing adaptersbased on, for example, one or more criteria that are particularized tothose devices. For example, at a given time a first assistant clientdevice can be preemptively executing adapters A, B, C, and D - and asecond assistant client device can be preemptively executing adapters C,D, E, and F.

In some implementations, a generic smart device control command isgenerated responsive to voice input, touch input, triggering of anautomated assistant routine, a signal from another non-assistant device,and/or other signal(s). For example, a generic smart device controlcommand identified at a given assistant client device can be generatedresponsive to voice input received at the given assistant client device.For instance, the given assistant client device can transmit audio data,corresponding to the voice input, to a remote assistant server, andreceive, in response, the generic smart device control command (e.g.,the remote assistant server can process the voice input to determine itcorresponds to the generic smart device control command). Also, forinstance, the given assistant client device can itself process the voiceinput (e.g., using a local voice-to-text processor and/or naturallanguage understanding engine) to locally generate the generic smartdevice control command. As another example, a generic smart devicecontrol command identified at a given assistant client device can begenerated responsive to touch input received at the given assistantclient device. For example, the given assistant client device canrender, on a touch sensitive screen thereof, an interactive graphicalinterface element for control of smart device(s) (e.g., an on/offelement, a dimmer element for smart light(s), a temperature adjustmentelement for smart thermostat(s), etc.) and can directly interpretinteractions with the interactive graphical interface element intocorresponding generic smart device control commands. As yet anotherexample, the generic smart device control command can be generated(locally or remotely) responsive to triggering of an automated assistantroutine that includes the generic smart device control command. Theautomated assistant routine can be triggered responsive to detection ofvoice input that is assigned to the routine (e.g., a shortcut phrasesuch as “good morning” that triggers a plurality of discrete actionsincluding smart device control action(s)), interaction with a virtual orphysical button for the routine, occurrence of certain temporalcondition(s), and/or occurrence of other triggering condition(s).

In some implementations, one or more criteria are utilized to select aparticular assistant client device, from an ecosystem of availableclient devices, for generating a specific control command and/or fortransmitting the specific control command to corresponding smartdevice(s). In some of those implementations, the particular assistantclient device is selected based on a reliable communications channelbeing established between the particular assistant client device and thecorresponding smart device(s). In some versions of thoseimplementations, the particular assistant client device can be selectedbased on the communications channel of the particular assistant clientdevice having at least a threshold signal strength with thecorresponding smart device(s). For example, the threshold signalstrength can be a fixed signal strength, or a signal strength relativeto signal strengths of other assistant client devices of the ecosystem.For instance, the particular client device can be selected to control asmart device based on a signal strength, between the particular clientdevice and the smart device via a communications channel for controllingthe smart device, being greatest amongst all devices of the ecosystem.

As one example, a generic smart device control command can be identifiedinitially at a first assistant client device, and the generic smartdevice control command can specify a particular smart device to becontrolled. The first assistant client device can determine that areliable communications channel is not established between it and theparticular smart device. For example, the particular smart device can beone to be controlled via a Bluetooth communications channel and thefirst assistant client device can determine that the particular smartdevice is not detectable via its Bluetooth radio, or that the signalstrength between the first assistant client device and the particularsmart device, via the Bluetooth radio, fails to satisfy a threshold. Inresponse, the first assistant client device can transmit the genericsmart device control command, over a local network (e.g., Wi-Fi or amesh network established between assistant client devices) to a secondassistant client device that has an established reliable communicationschannel to the smart device. The second assistant client device can thenuse a corresponding local adapter to translate the generic smart devicecontrol command to a corresponding specific command, and transmit thecorresponding specific command to the particular smart device.Alternatively, the first assistant client device can utilize itscorresponding local adapter to generate the corresponding specificcommand, and transmit the corresponding specific command to the secondassistant client device, in lieu of transmitting the generic smartdevice control command.

Continuing with the example, the first assistant client device canoptionally rely on a client devices to smart devices mapping, storedlocally at the assistant client device, to resolve the second assistantclient device as one having an established reliable communicationschannel to the smart device. The client devices to smart devices mappingcan define client devices of the ecosystem and, for each of theassistant client devices, can define corresponding smart device(s) forwhich the assistant client device has a corresponding reliablecommunications channel, and can optionally define signal strength(s)between the assistant client device and smart device(s). The mapping canbe generated based on corresponding data (e.g., detected device(s),signal strength(s)) reported by the assistant client devices, and can beupdated at periodic and/or non-periodic intervals. Remote automatedassistant component(s) can optionally receive the corresponding datafrom the assistant client devices of the ecosystem, update the mapping,and transmit the updated mappings to the assistant client devices of theecosystem which locally store and locally utilize the mappings. Thefirst assistant client device can initially identify the generic smartdevice control command based on, for example, corresponding user inputbeing received at the first assistant client device and/or the genericsmart device control command being generated locally at the firstassistant client device. In some implementations where the generic smartdevice control command is generated by remote automated assistantcomponent(s), such component(s) can access a client devices to smartdevices mapping, and initially address the generic smart device controlcommand to the second assistant client device, thereby causing it to beprovided directly to the second assistant client device, without beingrouted through the first assistant client device.

Various implementations described herein additionally or alternativelyrelate to utilizing local assistant client devices in discovering,provisioning, and/or registering smart devices for an account of a user.In some implementations, a smart device that is not yet registered canbe discovered by causing each assistant client device, of an ecosystemof assistant client devices, to scan one or more communications channels(e.g., Wi-Fi, Bluetooth, and/or other) for smart device(s) that aren’tregistered. For example, for Wi-Fi, each of the assistant client devicescan scan a respective Wi-Fi radio to identify any unregistereddevice(s), optionally using service set identifiers (SSIDs) and/or basicservice set identifiers (BSSIDs) to filter for particular smart devicesfor which corresponding adapters are available. As another example, forBluetooth, each of the assistant client devices can scan a respectiveBluetooth radio to identify any un-paired smart device(s), optionallyutilizing identifier(s) to filter for particular smart devices for whichcorresponding adapters are available. The discovery can occur atperiodic or non-periodic intervals, or in response to a user request(e.g., a voice request, a request initiated via a smart phone app forthe assistant, etc.).

Once discovered, a smart device can be provisioned by causing the smartdevice to pair with at least one of the assistant client devices (e.g.,in the case of Bluetooth) and/or to connect to a secured Wi-Fi networkto which the assistant client devices are already connected. Afterprovisioning, data transmitted by the smart device can be received, atan assistant client device, and processed using a local adapter of theassistant client device, to generate registration data in a schema ofthe automated assistant. For example, the data transmitted by the smartdevice can be in a protocol suite for a corresponding 3P, and can beinterpreted by the 3P adapter into a schema for registration with theautomated assistant. The registration data can be utilized by theautomated assistant to add the smart device to a home graph or otherstructure that defines smart devices, assistant client devices, andvarious properties (e.g., name(s), location(s), etc.) of the smartdevices and assistant client devices. Optionally, user input can bereceived via the assistant client device and/or other client device andcan be utilized to supplement the registration data. For example, theuser input can include one or more user provided aliases for the smartdevice, to enable storing of those alias(es) in association with thesmart device in the home graph.

In some implementations, the local adapter utilized to generate theregistration data is specific to the party of the smart device, and isoptionally downloaded at the assistant device responsive to thediscovery and/or provisioning of the smart device. In someimplementations, the smart device is assigned to a home graph (or otherstructure), based on biometric data, detected via one or more of theassistant devices of the ecosystem, corresponding to an accountassociated with the home graph. For example, the smart device can beassigned to the home graph (e.g., in lieu of another home graphassociated with another account of one or more of the assistant clientdevices) based on voice data, received during the discovering,provisioning, and/or registering, matching voice authentication datastored in association with the account. For example, the voice data canbe based on spoken input provided by the user in requesting a scanand/or in providing alias(es) for a discovered device. Other biometricidentification can additionally or alternatively be utilized, such asfacial, fingerprint, etc. In some implementations, a registered smartdevice can be associated (e.g., in a home graph) with those assistantclient device(s) that can locally transmit commands to the smart device.For example, if communications with the smart device occur via ashort-range radio (e.g., Bluetooth radio), those assistant clientdevices having at least a corresponding signal strength (for thecorresponding connection to the smart device via the short-range radio),can be designated in the home graph, optionally along with a designationof the corresponding signal strengths. Such designations can be used indetermining which assistant client device(s) should be utilized ingenerating and/or transmitting specific control commands to the smartdevice.

In some implementations, a registered smart device can be automatically(e.g., without any user input, or only confirmatory user input)associated (e.g., in a home graph) with a location that is based onlocation(s) of assistant client device(s) and signal strength(s) betweenthose assistant client device(s) and the registered smart device. As oneexample, if a signal strength between the registered smart device and agiven assistant client device, having a location of “office” in a homegraph, satisfies a threshold and/or is the “strongest” signal strength(amongst all assistant client devices), the registered smart device canbe automatically assigned a location of “office” in the home graph. Asanother example, if the registered smart device has signal strength(s),between it and assistant client device(s), that match (e.g., are withina threshold percentage of) signal strength(s) of other smart device(s)having a “laundry room” location in the home graph, the registered smartdevice can be automatically assigned a location of “laundry room” in thehome graph. For instance, assume an existing smart device in the homegraph has a manually labeled location of “laundry room”, a first signalstrength between it and a first assistant client device, and a secondsignal strength between it and a second assistant client device. In suchan instance, a newly registered smart device that has a third signalstrength between it and the first assistant client device that matches(e.g., within 5% of, 10% of, 15% of, or other threshold of) the firstsignal strength and a fourth signal strength that matches (e.g., within5% of 10% of, 15% of, or other threshold of) the second signal strength,it can be assumed to have the same “laundry room” location of theexisting smart device, and the “laundry room” location automaticallyassigned in the home graph to the newly registered smart device.Additionally or alternatively, in various implementation(s) signalstrength(s) for a smart device can be periodically (or at other regularor non-regular interval(s)) analyzed to determine whether a location ofthe smart device should be updated in the home graph (or other topologyrepresentation). For example, if a smart plug is moved from an “office”to a “living room”, such change can be automatically detected andupdated. Also, for example, if a battery powered smart device, or apassive smart device is readily movable about an environment, itslocation changes can be automatically detected and updated. In these andother manners, a location of a smart device can be automaticallydetermined and assigned in a home graph. This can enable the smartdevice to be controllable responsive to location-specific spokenutterances (e.g., “turn off all devices in the office”), can enablecontrol(s) to be rendered in appropriate locations in a visual interfacefor control (e.g., placing control(s) for the smart device in an“office” subheading or control group), and/or can enable a location ofthe smart device to be included in response to certain spoken utterances(e.g., “where is battery powered smart device A located”?).

In some implementations, a smart device is discovered that isbroadcasting an access point and that smart device is not yetprovisioned as it is not connected to a secured Wi-Fi network utilizedby the assistant client devices, and lacks the credentials forconnecting to the secured Wi-Fi network. In some of thoseimplementations, each of a plurality of assistant client devices, of theecosystem of client devices, detected the availability of the accesspoint through scanning of its respective Wi-Fi radio. In some versionsof those implementations, a single assistant client device is selectedfor connecting to the smart device via the access point of the smartdevice, and securely sharing, with the smart device via the connection,the credentials for connecting to the secured Wi-Fi network. Once thecredentials are shared, the smart device connects to the secured Wi-Finetwork and can be registered through communications via the Wi-Finetwork. In various versions where the single assistant client device isselected, it is selected based at least in part on human interactiondata for the single assistant client device. For example, it can beselected based on the human interaction data indicating that no userinterface input has been received at the single assistant client devicewithin a threshold amount of time and/or based on the human interactiondata indicating that the single assistant client device is utilized lessfrequently than any other of the plurality of assistant client devicesthat detected the access point. In these and other manners, the singleclient device can be selected so that disruption to ongoing and/orforthcoming assistant interactions is prevented, as the single clientdevice will break its connection with the secured Wi-Fi network whenconnecting to the local access point - and the single client device isselected based on criteria that indicated it is not being used and/or isless likely (than other devices of the ecosystem) to be used.

The above description is provided as an overview of only someimplementations of the present disclosure. Further description of thoseimplementations, and other implementations, are described in more detailherein.

In addition, some implementations include one or more processors of oneor more computing devices, where the one or more processors are operableto execute instructions stored in associated memory, and where theinstructions are configured to cause performance of any of the methodsdescribed herein. Some implementations include a client device with oneor more processors executing locally stored instructions and interfacingwith locally stored data to perform one or more of the methods describedherein. Some implementations also include one or more non-transitorycomputer readable storage media storing computer instructions executableby one or more processors to perform any of the methods describedherein.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts described in greater detail herein arecontemplated as being part of the subject matter disclosed herein. Forexample, all combinations of claimed subject matter appearing at the endof this disclosure are contemplated as being part of the subject matterdisclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example environment in whichimplementations disclosed herein may be implemented.

FIG. 2 illustrates an example environment that includes multiple smartdevices, from a plurality of disparate parties, and that includesmultiple assistant devices, each of which can locally control one ormore of the smart devices according to implementations disclosed herein.

FIG. 3 illustrates a state diagram of an example of controlling twosmart light bulbs of FIG. 2 , using one of the client devices of FIG. 2.

FIG. 4 illustrates a state diagram of an example of routing a genericsmart device control command, for controlling another light bulb of FIG.2 , from a first client device of FIG. 2 to an alternate client deviceof FIG. 2 .

FIG. 5 is a flow chart illustrating an example method of controlling asmart device, using a local assistant client device, according tovarious implementations disclosed herein.

FIG. 6 is a flow chart illustrating another example method ofcontrolling a smart device, using a local assistant client device,according to various implementations disclosed herein.

FIG. 7 is a flow chart illustrating an example method of registering asmart device, using a local assistant client device, according tovarious implementations disclosed herein.

FIG. 8 illustrates an example architecture of a computing device.

DETAILED DESCRIPTION

There is a proliferation of smart network connected devices (alsoreferred to herein as smart devices or Internet of Things (IoT) devices)such as smart home alarms, smart door locks, smart cameras, smartlights, smart thermostats, smart weight scales, smart beds, smartirrigation systems, smart garage door openers, smart plugs, smartappliances, smart baby monitors, smart fire alarms, smart moisturedetectors, etc. Often, multiple smart devices are located within theconfines of a structure, such as a home - or located within multiplerelated structures, such as a user’s primary residence and the user’ssecondary residence and/or work location.

Further, there is a proliferation of assistant client devices that eachinclude an assistant client that can optionally interact with one ormore remote automated assistant components to form a logical instance ofan automated assistant. An assistant client device can be devoted solelyto assistant functionality (e.g., a standalone speaker and/or standaloneaudio/visual device including only an assistant client and associatedinterface, and devoted solely to assistant functionality) or can performassistant functionality in addition to other functions (e.g., a mobilephone or tablet that includes an assistant client as one of multipleapplications). Moreover, some smart devices can also be assistant clientdevices. For example, some smart devices can include an assistant clientand at least speaker(s) and/or microphone(s) that serve (at least inpart) as user interface output and/or input devices for an assistantinterface of the assistant client.

Various techniques have been proposed for associating smart devices withcorresponding logical instances of automated assistants (and optionallywith individual assistant client devices). For example, a user, group ofusers, an assistant client device, and/or a group of assistant clientdevices (e.g., all within a structure) can be linked (e.g., in one ormore databases) with a plurality of disparate smart devices to enableinteraction with (e.g., control of) the smart devices via automatedassistants. For instance, each of multiple assistant client devices in ahousehold can be linked to each of multiple disparate smart devices inthe household to enable any user (or a restricted group of users) tointerface with any one of the assistant client devices to interact withany one of the multiple disparate smart devices.

One example of such linking is a device topology representation (alsoreferred to herein as a “home graph”) that can be user created, and/orautomatically created, and that may define various assistant clientdevices, various smart devices, identifier(s) for each, and/orattribute(s) for each. For example, the identifier(s) for a device canspecify a room (and/or other area(s)) of a structure in which the deviceis located (e.g., living room, kitchen) and/or can specify nickname(s)and/or alias(es) for the device (e.g. couch lamp, front door lock,bedroom speaker, kitchen assistant, etc.). In this manner, theidentifiers of devices can be names, aliases, and/or locations of therespective devices that the user is likely to associate with therespective devices.

The device topology representation can further specify one or moredevice attributes associated with the respective devices. The deviceattributes for an assistant client device can be, for example,associated with one or more input and/or output modalities supported bythe assistant client device. For instance, a device attribute for astandalone speaker-only assistant client device can indicate that it iscapable of providing audible output, but incapable of providing visualoutput. The device attributes of a smart device can, for example,identify one or more states, of the smart device, that can becontrolled; identify a party (e.g., a 3P) that manufactures,distributes, and/or creates the firmware for the smart device; and/oridentify a unique identifier for the smart device, such as a 1P or 3Pprovided fixed identifier. According to various implementationsdisclosed herein, the device topology representation can optionallyfurther specify: which smart devices can be controlled locally by whichassistant client devices; local addresses for locally controllable smartdevices (or local addresses for hubs that can directly locally controlthose smart devices); local signal strengths and/or other preferenceindicators between assistant client devices and smart devices. Further,according to various implementations disclosed herein, the devicetopology representation (or a variation thereof) can be locally storedat each of a plurality of assistant client devices for utilization inlocally controlling and/or locally registering smart devices.

Now turning to FIG. 1 , an example environment in which techniquesdisclosed herein may be implemented is illustrated. The exampleenvironment includes a plurality of client computing devices 110 _(1-N)(also referred to herein simply as “client devices”), cloud-basedautomated assistant components 120, smart device systems 140 _(A-N),smart devices 145 _(A-N), 3P adapters database 150, and a home graph 152for a user of the client devices 110 _(1-N). The client devices 110_(1-N) and smart devices 145 _(1-N) of FIG. 1 represent client devicesand smart devices that are at least selectively associated with oneanother (e.g., via the home graph 152 and/or other device topology). Forexample, the smart devices 145 _(1-N) can all be at a home (e.g., in theinterior and/or exterior of the home), the client devices 110 _(1-N) canbe at least occasionally in the same home, and the smart devices 145_(1-N) and the client devices 110 _(1-N) can be linked to one anotherutilizing one or more techniques, such as those described herein.Through such linking, the client devices 110 _(1-N) can be utilized tolocally control the smart devices 145 _(1-N) according toimplementations described herein.

One or more (e.g., all) of the client devices 110 _(1-N) can execute arespective instance of an automated assistant client 117 _(1-N).However, in some implementations one or more of the client devices 110_(1-N) can optionally lack an instance of an automated assistant client117 _(1-N), and still include engine(s) and hardware components forlocally controlling and/or registering one or more smart devices (e.g.,an adapter interaction engine, adapter(s), radio(s), routing engine,and/or a registration engine). An instance of the automated assistantclient 117 _(1-N) can be an application that is separate from anoperating system of the corresponding client device 110 _(1-N) (e.g.,installed “on top” of the operating system) - or can alternatively beimplemented directly by the operating system of the corresponding clientdevice 110 _(1-N). As described further below, each instance of theautomated assistant client 117 _(1-N) can optionally interact withcloud-based automated assistant component(s) 120 in responding tovarious requests provided by a user via I/O components 111 of any one ofthe client devices 110 _(1-N). Further, and as also described below,other engine(s) of the client devices 110 _(1-N) can optionally interactwith cloud-based automated assistant component(s) 120.

One or more cloud-based automated assistant components 120 can beimplemented on one or more computing systems (collectively referred toas a “cloud” or a “remote” computing system) that are communicativelycoupled to client devices 110 _(1-N) via one or more wide area networks(e.g., the Internet), indicated generally by 105 ₁ of FIG. 1 . Forexample, cloud-based automated assistant component(s) 120 can beimplemented by one or more clusters of high-performance servers. It isnoted that the client devices 110 _(1-N) can utilize one or more localarea networks in accessing the wide-area networks 105 ₁ and/or inlocally communicating with one another. Such local area networks caninclude a Wi-Fi network and/or a mesh network between the client devices110 _(1-N).

The cloud-based automated assistant components 120 can also becommunicatively coupled with smart device systems 140 _(A-N) via one ormore wide area networks. The communicative coupling of the cloud-basedautomated assistant components 120 with the smart device systems 140 isindicated generally by 110 ₂ of FIG. 1 . Further, the smart devicesystems 140 can each be communicatively coupled to a corresponding groupof one or more smart devices 145 _(A-N) via one or more wide areanetworks, generally indicated generally by 110 ₄ of FIG. 1 . It is notedthat the smart devices 145 _(A-N) can utilize one or more local areanetworks in accessing the wide-area networks 105 ₃.

Each of the smart device systems 140 _(A-N) can be either a 1P or a 3Psystem, and each can be communicatively coupled with one or morecorresponding smart devices 145 _(A-N). For example, a first smartdevice system 140 _(A-N) can be controlled by a first 3P andcommunicatively coupled with a first smart device 145 _(A1), a secondsmart device system 140 can be controlled by a second 3P andcommunicatively coupled with a second smart device 145 _(B1) and a thirdsmart device 145 _(B2), etc.

The smart device systems 140 can communicate with the devices 145 _(A-N)via the wide-area networks 105 ₃ to control their respective smartdevices 145 _(A-N), to deliver firmware updates to their respectivesmart devices 145 _(A-N), to ascertain the status of their respectivesmart devices 145 _(A-N), etc. For example, a given one of the smartdevice systems 140 can communicate with one of the smart devices 145_(A-N) to control the smart device in response to user input beingreceived via a mobile application, for the smart device system, thatenables control of the smart device.

Also, for example, a given one of the smart device systems 140 cancommunicate with one of the smart devices 145 _(A-N) to control thesmart device in response to a request from cloud-based automatedassistant component(s) 120. For example, according to some techniques auser can provide, via one or more I/O components 111 ₁ of a clientdevice 110 ₁, a request to control a smart device, such as spoken inputof “turn off the couch light” provided via microphone(s) of I/Ocomponents 111 ₁. The request (e.g., the spoken input or a conversionthereof) can be transmitted by the automated assistant client 117 ₁ ofthe client device 110 ₁, to the cloud-based automated assistantcomponent(s) 120 via the wide-area networks 105 ₁. The cloud-basedautomated assistant component(s) 120 can process the request todetermine a smart device to be controlled based on the request, andtransmit, via the wide-area networks 105 ₂, a request to a respectiveone of the smart device systems 140 _(A-N) which, in turn transmits, viawide-area networks 105 ₃, corresponding command(s) to the smart device.However, as described herein such techniques present drawbacks such ashigh latency, low reliability, and/or excessive consumption of networkresources.

While implementations disclosed herein can still utilize such techniquesfor some smart device control requests and/or in some situations,various implementations disclosed herein can at least selectivelybypass, in controlling a smart device, communications between thecloud-based automated component(s) 120 and a respective one of the smartdevice system(s) 140 and/or communication(s) between the smart devicesystem(s) 140 and the smart device. Rather, those variousimplementations generate, locally at one of the client devices 110_(1-N), specific control commands that are transmitted locally (e.g.,via a local-area-network or a local peer-to-peer connection) to smartdevice(s) to effectuate a desired state change at the smart device(s).This can mitigate the high latency, low reliability, excessiveconsumption of network resources, and/or other drawbacks of theabove-mentioned techniques.

In some implementations, the plurality of client computing devices 110_(1-N) and smart devices 145 _(A-N) can be associated with each other invarious ways in order to facilitate performance of techniques describedherein. For example, in some implementations, the plurality of clientdevices 110 _(1-N) and smart devices 145 _(A-N) may be associated witheach other by virtue of being communicatively coupled via one or moreLANs and/or via one or more peer-to-peer networks. This may be the case,for instance, where plurality of client computing devices 110 _(1-N) aredeployed across a particular area or environment, such as a home, abuilding, and so forth. Additionally or alternatively, in someimplementations, plurality of client devices 110 _(1-N) and smartdevices 145 _(A-N) may be associated with each other by virtue of thembeing members of a coordinated ecosystem of client devices 110 _(1-N)and smart devices 145 _(A-N) that are at least selectively accessible byone or more users (e.g., an individual, a family, employees of anorganization, other predefined groups, etc.). In some of thoseimplementations, the ecosystem of client devices 110 _(1-N) and smartdevices 145 _(1-N) can be manually and/or automatically associated witheach other in the home graph 152 and/or other device topology. Forexample, one or more of the client devices 110 _(1-N) and smart devices145 _(1-N) can be associated with one another through registrationtechniques described herein.

An instance of an automated assistant client 117 _(1-N), by way of itsinteractions with one or more cloud-based automated assistant components120, may form what appears to be, from a user’s perspective, a logicalinstance of an automated assistant with which the user may engage in ahuman-to-computer dialog. For example, a user can engage with the samelogical instance of an automated assistant using either client device110 ₁ and automated assistant client 117 ₁ or using client device 110_(N) and automated assistant client 117 _(N). While the particularinstances of the automated assistant client 117 ₁ and 117 _(N) can varyand/or can provide user interface output via different I/O components111 ₁ and 111 _(N) and/or accept different user interface input viadifferent I/O components 111 ₁ and 111 _(N) (e.g., I/O components 111 ₁can include a touch-screen, while I/O components 111 _(N) do not), theuser may still effectively engage with the same logical instance of theautomated assistant. For the sakes of brevity and simplicity, the term“automated assistant”, as used herein will refer to the automatedassistant client 117 executing on a client device 110 and optionally toone or more cloud-based automated assistant components 120 (which may beshared amongst multiple automated assistant clients 117). Although twoclient devices 110 ₁ and 110 _(N) of a coordinated ecosystem areillustrated in FIG. 1 , it is understood that many additional clientdevices can be included in the ecosystem. Further, it is understood thatseparate coordinated ecosystems of client devices will also be provided,each associated with different user(s) (e.g., account(s)) and/orenvironments, and that such separate coordinated ecosystems can alsointeract with cloud-based automated assistant component(s) 120 (but withinteractions tailored to the account(s) of those separate ecosystems).

The client devices 110 _(1-N) may include, for example, one or more of:a desktop computing device, a laptop computing device, a tabletcomputing device, a mobile phone computing device, a computing device ofa vehicle of the user (e.g., an in-vehicle communications system, anin-vehicle entertainment system, an in-vehicle navigation system), astandalone assistant-centric interactive speaker, a standaloneassistant-centric interactive display with speaker(s), a smart appliancesuch as a smart television, and/or a wearable apparatus of the user thatincludes a computing device (e.g., a watch of the user having acomputing device, glasses of the user having a computing device, avirtual or augmented reality computing device). Additional and/oralternative client computing devices may be provided.

As mentioned above, one or more of the client devices 110 _(1-N) canoptionally include a respective instance of an automated assistantclient 117 _(1-N). The automated assistant clients 117 _(1-N) can eachprocess user inputs received from input device(s) of respective I/Ocomponents 111 _(1-N), such as spoken inputs detected via microphone(s)of I/O components 111 _(1-N), touch inputs received via touch-screendisplays of I/O components 111 _(1-N), images detected via camera(s) ofI/O components 111 _(1-N), etc. Further, each of the automated assistantclients 117₁₋ _(N) can optionally render various outputs via outputdevice(s) of respective I/O components 111 ₁₋ _(N), such as speaker(s)and/or touch-screen displays of I/O components 111 _(1-N).

In various implementations, one or more instances of the automatedassistant client 117 _(1-N) can interface with the cloud-based automatedassistant component(s) 120 in processing inputs and/or in generatingoutputs based on the inputs and/or in generating smart device controlcommands based on the inputs. For example, automated assistant client117 ₁ can transmit, to cloud-based automated assistant components 120,audio data corresponding to spoken input received after invocation ofthe automated assistant client 117 ₁ at the client device 110 ₁. Theinvocation of the automated assistant client 117 ₁ at the client device110 ₁ can be based on detecting an invocation phrase (e.g., “OKAssistant”), interaction of a hardware button or graphical button thatinvokes the automated assistant client 117 ₁, in response to a gesturedetected via a camera of the I/O components 111 ₁, and/or otherinvocation signal(s). The cloud-based automated assistant components 120can convert the audio data to text using speech-to-text (STT) processor121, and perform natural language understanding (NLU) on the text usingNLU engine 122 to determine an appropriate response. For example, theappropriate response can be a textual response that can optionally beconverted to generated speech using text-to-speech (TTS) processor, andtransmitted to the automated assistant client 117 ₁ for rendering of thegenerated speech via speaker(s) of the I/O components 111 ₁.

As another example, the appropriate response can be a generic smartdevice control command, such as a structured command that defines anintent (e.g., turn on/off, dim, increase, decrease), defines a smartdevice (e.g., a unique identifier of the smart device), defines athird-party for the smart device (e.g., a unique identifier for thethird-party), and/or one or more further parameters for the intent(e.g., how much to dim, what to increase/decrease, etc.). The genericsmart device control command can be transmitted to the assistant client117 ₁, or another assistant client of another client device 110, forlocal implementation of the generic smart device control command. Insome implementations and/or for some assistant clients 117 _(1-N), theassistant client can include a local STT processor, local NLU engine,and/or other component(s) for locally processing received inputs,without necessitating interfacing with cloud-based automated assistantcomponent(s) 120. For example, assistant client 117 ₁ can include one ormore local component(s) for locally generating generic smart devicecontrol commands, without interfacing with cloud-based automatedassistant component(s) 120.

One or more of the client devices 110 also optionally includes arespective adapter interaction engine 114 _(1-N), one or more respectivelocally stored adapters 113 _(A-N), and one or more respective availableradios 112 _(A-N). The same adapters 113 _(A-N) and available radios 112_(A-N) can optionally be locally provided on each of the client devices110 _(1-N), or the adapters 113 _(A-N), and/or radios 112 _(A-N) canoptionally vary among client devices 110 _(1-N). Further, as describedherein, one or more of the adapters 113 _(A-N) can be preemptivelyexecuted at a client device 110 _(1-N), and which adapters arepreemptively executing at a given time for a given client device110_(1-N)can be based on one or more criteria, such as historical usageof the client device, temporal criteria, etc.

Further description of adapter interaction engines 114 _(1-N) andlocally stored adapters 113 _(A-N) is now provided with reference toclient device 110 ₁. A generic smart device control command can bepassed from the automated assistant client 117 ₁ to the adapterinteraction engine 114 ₁. Although illustrated separately in FIG. 1 forclarity, it is noted that in various implementations adapter interactionengine 114 ₁, adapters 113 _(A-N), routing engine 115 ₁, and/orregistration engine 116 ₁ can logically be incorporated as part ofautomated assistant client 117 ₁. The generic smart device controlcommand can be generated responsive to voice input, touch input,triggering of an automated assistant routine, a signal from anothernon-assistant device, and/or other signal(s). For example, the genericsmart device control command can be generated responsive to voice inputreceived at the client device 110 ₁. For instance, the automatedassistant client 117 ₁ can transmit audio data, corresponding to thevoice input, to remote automated assistant component(s) 120, andreceive, in response, the generic smart device control command. Also,for instance, the automated assistant client 117 ₁ can itself processthe voice input to locally generate the generic smart device controlcommand. As another example, automated assistant client 117 ₁ canrender, on a touch sensitive screen of I/O components 111 ₁, aninteractive graphical interface element for control of smart device(s)and can directly interpret interactions with the interactive graphicalinterface element into corresponding generic smart device controlcommands. The generic smart device control command can be a structuredcommand that defines an intent, defines smart device(s) to becontrolled, optionally defines a third-party for the smart device,and/or one or optionally one or more further parameters for the intent.

The adapter interaction engine 114 ₁ can select, based on the genericsmart device control command, a corresponding one of the adapters 113_(A-N) based on that adapter being provided by a party that correspondsto the generic smart device control command. The adapter interactionengine 114 ₁ provides the generic smart device control command to theselected one of the adapters 113 _(A-N), which processes the genericsmart device control command to generate specific control commands thatare directly interpretable by corresponding smart device(s) toeffectuate state change(s) at the corresponding smart device(s), or atcorresponding additional smart device(s) directly controlled by thecorresponding smart device. In various implementations, each of adapters113 _(A-N) is implemented by JavaScript (or other interpretedprogramming language) provided by a party corresponding to the adapter,and each of the adapters 113 _(A-N) can translate generic smart devicecontrol commands into specific control commands that conform to aprotocol suite of the party. Each of adapters 113 _(A-N) can besandboxed, executing within a container within the client device 110 ₁or the automated assistant client 117 ₁, can process generic smartdevice control commands as described, and can further optionally reportevents to the automated assistant client 117 ₁ using the schema of theautomated assistant. The adapters 113 _(A-N) can be transmitted to theclient device 110 ₁ by the cloud-based automated assistant component(s)120, which can select the adapters 113 _(A-N) from 3P adapters database150. The adapters of 3P adapters database 150 can be updated by smartdevice systems 140 _(A-N) (e.g., to conform to firmware updates and/orto improve functionality) and updated adapters pushed to the clientdevice 110 ₁. In some implementations, additional adapter(s) can beprovided and utilized to process certain generic smart device controlcommands, optionally for certain smart device(s). For example, a givenadditional adapter can be utilized to process certain generic smartdevice control commands, and generate corresponding specific controlcommands that are common for multiple 3P protocol suites (e.g., commonto an industry standard base protocol to which the 3P protocol suite(s)conform or are variations). For instance, a generic wake-on-lan (WOL)command can be processed using the given additional adapter to generatea specific WOL control command that is applicable to all protocol suitesthat are variation(s) of the IP protocol industry standard. In these andother manners, additional adapter(s) can be utilized to process certaingeneric smart device control command(s) (e.g., for smart devices ofmultiple 3Ps), while 3P specific adapter(s) can be utilized to processcertain other generic smart device control command(s).

The generated specific control commands are provided to the adapterinteraction engine 114 ₁. The adapter interaction engine 114 ₁ selectsone of the radios 112_(A-N)for locally transmitting the specific controlcommand to effectuate alteration of state(s) of the smart device(s)indicated by the specific control command (and the generic smart devicecontrol command on which the specific control command is based). Theadapter interaction engine 114 ₁ can select the radio based on it beingassigned, locally, to the party and/or adapter corresponding to thegeneric smart device control command. The adapter interaction engine 114₁ then causes the specific control commands to be transmitted via theselected one of the radios 112 _(A-N), and addressed to the smartdevice(s), to thereby effectuate a state change at the addressed smartdevice(s), or at smart device(s) connected to the addressed smartdevice(s) (e.g., when the addressed smart device(s) is a hub controllingadditional smart device(s)).

One or more of the client devices 110 further optionally include arespective routing engine 115 _(1-N) and/or a respective registrationengine 116 _(1-N). Each routing engine 115 _(1-N) can selectivelydetermine to route, via a local network, a received generic smart devicecontrol command to an alternate client device, of the ecosystem ofclient devices 110 _(1-N), based on one or more criteria. For example,the client device 110 ₁ can initially identify a smart device controlcommand based on, for example, the smart device control command beinggenerated locally at the client device 110 ₁ (e.g., responsive totouch-interaction with a corresponding graphical element displayed atthe client device 110 ₁) or being generated at remote automatedassistant components 120, and initially being provided to the clientdevice 110 ₁ based on being generated based on spoken or other inputreceived at the client device 110 ₁.

The generic smart device control command can specify a particular smartdevice to be controlled, and the routing engine 115 ₁ can determine thata reliable communications channel is not established between the clientdevice 110 ₁ and the particular smart device. For example, theparticular smart device can be one to be controlled via a Bluetoothcommunications channel and the routing engine 115 ₁ can determine thatthe particular smart device is not detectable via a Bluetooth radio ofradios 112 _(A-N) of client device 110 ₁, or a signal strength betweenthe client device 110 ₁ and the particular smart device, via theBluetooth radio, fails to satisfy a threshold. Such a determination canbe based on, for example, current readings via the Bluetooth radioand/or based on local version of the home graph 152 ₁ indicating thatthe client device 110 ₁ does not have a reliable connection with theparticular smart device.

In response, the routing engine 115 ₁ can transmit the generic smartdevice control command, over a local network (e.g., Wi-Fi or a meshnetwork established between assistant client devices 110 _(1-N)) to analternate client device 110 _(2-N) that has an established reliablecommunications channel to the smart device. The routing engine 115 ₁ candetermine such an alternate client device utilizing, for example, localversion of the home graph 152 ₁, which can indicate that the alternateclient device has a reliable connection with the smart device. Forexample, the routing engine 115 ₁ can select the alternate client devicebased on the local version of the home graph 152 ₁ indicating thealternate client device has the best signal strength, for the smartdevice, amongst all client devices 110 _(1-N). The local version of thehome graph 152 ₁ is locally stored at the client device 110 ₁ and can bethe same or a variant of the home graph 152 maintained by the remoteautomated assistant component(s) 120 (e.g., a compressed version, aslightly less up to date version). The alternate client device, inresponse to receiving the generic smart device control command, can thenuse a corresponding local adapter to translate the generic smart devicecontrol command to a corresponding specific command, and transmit thecorresponding specific command to the particular smart device.Alternatively, the adapter interaction engine 114 ₁ can generate thecorresponding specific command using a corresponding local one of theadapters 113 _(A-N), and transmit the corresponding specific command tothe alternate client device, in lieu of transmitting the generic smartdevice control command.

Each of the registration engines 116 _(1-N) is utilized in locallydiscovering, provisioning, and/or registering smart devices for anaccount of a user. Each of the registration engines 116 _(1-N) caninterface with a respective locally stored home graph 152 _(A-N),respective adapters 113 _(A-N) (e.g., via a respective adapterinteraction engine 114 _(1-N)), and respective radios 112 _(A-N) indiscovering, provisioning, and/or registering smart devices. Further,each of the registration engines 116 _(1-N) can interface with a cloudregistration engine 127 of cloud-based automated assistant component(s)120.

In some implementations, a smart device that is not yet registered canbe discovered by causing each of the registration engines 116 _(1-N) toscan one or more respective communications channels (e.g., Wi-Fi,Bluetooth, and/or other) for smart device(s) that aren’t registered. Forexample, for Wi-Fi, each of the registration engines 116 _(1-N) cancause a scan to be performed via a respective Wi-Fi radio, of radios 112_(A-N), to identify any unregistered device(s), optionally using serviceset identifiers (SSIDs) and/or basic service set identifiers (BSSIDs) tofilter for particular smart devices for which corresponding adapters areavailable. As another example, for Bluetooth, each of the registrationengines 116 _(1-N) can cause a scan to be performed via a respectiveBluetooth radio to identify any un-paired smart device(s), optionallyutilizing identifier(s) to filter for particular smart devices for whichcorresponding adapters are available. The discovery can occur atperiodic or non-periodic intervals, or in response to a user request(e.g., a voice request, a request initiated via a smart phone app forthe assistant, etc.).

When one or more of the registration engines 116 _(1-N) discovers asmart device, the routing engine of one of the smart devices thatdiscovered the smart device can provision the smart device by pairingwith it (e.g., in the case of Bluetooth) and/or causing it to beconnected to a secured Wi-Fi network to which the client devices 110_(1-N) are already connected. Which of the registration engines 116_(1-N) provisions the smart device can be determined based onnegotiation directly between the client devices 110 _(1-N) and/or can bedetermined by cloud routing engine 125, which can receive indications,from registration engines_(1-N), of which registration engines_(1-N)discovered the smart device.

After provisioning, data provided by the provisioned smart device can bereceived, at one of the client devices 110 _(1-N), and processed using acorresponding local adapter 113 _(A-N) of the assistant client device,to generate registration data in a schema of the automated assistant.The corresponding local adapter can already be locally provided at theclient device or can optionally be pushed, to the client device, bycloud-based automated assistant component(s) 120 responsive to theprovisioning of the smart device. For example, the cloud registrationengine 127 can access 3P adapters database 152 to retrieve a 3P adapterthat corresponds to the smart device that is provisioned, and push the3P adapter to one or more (e.g., all) of the client devices 110 _(1-N)for local storage and utilization at the client devices 110 _(1-N). Theregistration data generated by the local adapter 113 _(A-N) can beutilized by the automated assistant to add the smart device to a homegraph or other structure that defines smart devices, assistant clientdevices, and various properties (e.g., name(s), location(s), etc.) ofthe smart devices and assistant client devices. For example, theregistration data can be transmitted to the cloud registration engine127, which can add the registration data to the home graph 152, andoptionally push updated local versions of the home graph 152 _(1-N) toeach of the client devices 110 _(1-N). The cloud registration engine 127can optionally also interface with a corresponding one of the smartdevice systems 140 _(A-N), to report the registration to the smartdevice system to thereby enable the smart device system to locallycorrelate the smart device registration.

In some implementations, the cloud registration engine 127 registers asmart device with the home graph 152, based on biometric data, detectedvia one or more of the client devices 110 _(1-N), corresponding to anaccount associated with the home graph 152. For example, the smartdevice can be assigned to the home graph 152 (e.g., in lieu of anotherhome graph associated with another account of one or more of the clientdevices 110 _(1-N)) based on voice data, received via one or more of theclient devices 110 _(1-N) during the discovering, provisioning, and/orregistering, matching voice authentication data stored in associationwith the account. Other biometric identification can additionally oralternatively be utilized, such as facial, fingerprint, etc. In someimplementations, the cloud registration engine 127 can associate aregistered smart device (e.g., in home graph 152 and local versions)with those client device(s) 110 _(1-N) that can locally transmitcommands to the smart device. For example, if communications with thesmart device occur via a short-range radio (e.g., Bluetooth radio),those client devices 110 _(1-N) having at least a corresponding signalstrength (for the corresponding connection to the smart device via theshort-range radio), can be designated in the home graph 152, optionallyalong with a designation of the corresponding signal strengths. Suchdesignations can be used by routing engines 115 _(1-N) and/or by cloudrouting engine 125 in determining which assistant client device(s)should be utilized in generating and/or transmitting specific controlcommands to the smart device.

In some implementations, a smart device is discovered that isbroadcasting an access point and that smart device is not yetprovisioned as it is not connected to a secured Wi-Fi network utilizedby the client devices 110 _(1-N), and lacks the credentials forconnecting to the secured Wi-Fi network. In some of thoseimplementations, each of a plurality of the client devices 110 _(1-N),detected the availability of the access point through scanning of itsrespective Wi-Fi radio. In some versions of those implementations, asingle assistant client device is selected for connecting to the smartdevice via the access point of the smart device, and securely sharing,with the smart device via the connection, the credentials for connectingto the secured Wi-Fi network. The single client device can be selectedthrough negotiation of the registration engines 116 _(1-N) of the clientdevices, or by the cloud registration engine 127. Once the credentialsare shared, the smart device connects to the secured Wi-Fi network andcan be registered through communications via the Wi-Fi network. Invarious versions where the single assistant client device is selected,it is selected based at least in part on human interaction data for thesingle assistant client device. For example, it can be selected based onthe human interaction data indicating that no user interface input hasbeen received at the single assistant client device within a thresholdamount of time and/or based on the human interaction data indicatingthat the single assistant client device is utilized less frequently thanany other of the plurality of assistant client devices that detected theaccess point. In these and other manners, the single client device canbe selected so that disruption to ongoing and/or forthcoming assistantinteractions is prevented, as the single client device will break itsconnection with the secured Wi-Fi network when connecting to the localaccess point - and the single client device is selected based oncriteria that indicated it is not being used and/or is less likely (thanother devices of the ecosystem) to be used.

In some implementations, a smart device that is not yet registered canbe discovered by causing each of the registration engines 1161-N to scanone or more respective communications channels (e.g., Wi-Fi, Bluetooth,and/or other) for smart device(s) that aren’t registered. However, insome of those implementations the smart device may not be registrablelocally and/or controllable locally. Nonetheless, in some versions ofthose implementations a notification related to the discovered smartdevice can be presented at one or more of the assistant client device(s)and/or other client device(s). The notification can include link(s)and/or other interactive content that can be interacted with tofacilitate registration in a non-local manner (e.g., via the cloud). Forexample, the interactive content, when selected, can result in renderingof a single sign in interface and/or a smart device specific sign ininterface that enables registering of the smart device.

As mentioned above, cloud-based automated assistant component(s) 120 canoptionally interface with each of the client devices 110 in performanceof certain operations. The cloud-based automated assistant components120 can include a STT module 121 configured to leverage the resources ofthe cloud to convert audio data (provided by a corresponding clientdevice 110) into text (which may then be provided to NLU module 122).TTS module 123 may be configured to leverage the resources of the cloudto convert textual data into computer-generated speech output. NLUmodule 122 processes natural language input generated by users viaclient devices 110 _(1-N) and may generate annotated output for use byone or more other components of cloud-based automated assistantcomponent(s) 120. For example, the NLU module 122 may process naturallanguage free-form input that is generated by a user via one or moreuser interface input devices of client device 110 ₁, such as text thathas been converted from spoken input, or typed text. The generatedannotated output includes one or more annotations of the naturallanguage input and optionally one or more (e.g., all) of the terms ofthe natural language input.

In some implementations, the NLU module 122 is configured to identifyand annotate various types of grammatical information in naturallanguage input. For example, the NLU module 122 may include a part ofspeech tagger and/or an entity tagger. In some implementations, the NLUmodule 122 may additionally and/or alternatively include a coreferenceresolver (not depicted) configured to group, or “cluster,” references tothe same entity based on one or more contextual cues. In someimplementations, one or more components of the NLU module 122 may relyon annotations from one or more other components of the NLU module 122.

Cloud routing engine 125 can optionally be utilized to select aparticular client device, of client devices 110 _(1-N), to which toprovide a given generic smart device control command. The cloud routingengine 125 can utilize the home graph 152 in selecting the particularclient device. For example, the cloud routing engine 125 can utilize thehome graph 152 to select the particular client device based on the homegraph 152 indicating that the particular client device has anestablished communication channel with smart device(s) of the genericsmart device control command and/or has the best (e.g., highest signalstrength) communications channel with smart device(s) of the genericsmart device control command. The cloud routing engine 125 can thenaddress the generic smart device control command to the selectedparticular client device (e.g., using an address stored in the homegraph) to cause it to be transmitted directly to that client device, forlocally converting, at the selected particular client device, tospecific control commands and for locally transmitting, at the selectedparticular client device, the specific control commands to correspondingsmart device(s). The cloud routing engine 125 can route a generic smartdevice control command to a particular client device, even when thegeneric smart device control command is generated responsive to userinterface input received at a difference client device.

As described herein, in some implementations cloud-based automatedassistant component(s) 120 (e.g., STT module 121 and NLU module 122) canprocess audio data corresponding to spoken input captured at one of theclient devices 110, generate a generic smart device control commandbased on such spoken input, and return the generic command to the one ofthe client devices 110 for local generation, at the one of the clientdevices, of specific commands for the corresponding smart device(s). Inother implementations or situations, the cloud-based automated assistantcomponent(s) can provide generic smart device control command(s)responsive to other signals, such as responsive to triggering of anautomated assistant routine that involves the generic smart devicecontrol command.

Additional description of various components of FIG. 1 is now providedwith reference to the additional figures. FIG. 2 depicts a homefloorplan that includes a plurality of rooms, 250, 252, 254, 256, 258,260, and 262. A plurality of client devices 110 ₁₋₃ are deployedthroughout at least some of the rooms. Each of the client devices 110₁₋₃ can optionally implement an instance of an automated assistantclient configured with selected aspects of the present disclosure andcan optionally include one or more input devices, such as microphones,touch-screens, etc. and/or one or more output devices such as speakers,displays, etc. For example, a first client device 110 ₁ taking the formof a standalone interactive speaker and display device (e.g., displayscreen, projector, etc.) is deployed in room 250, which in this exampleis a kitchen. A second client device 110 ₂ taking the form of aninteractive standalone speaker is deployed in room 254, which in thisexample is a bedroom. A third client device 110 ₃, also taking the formof an interactive standalone speaker, is deployed in room 256.

The plurality of client devices 110 ₁₋₃ may be communicatively coupledwith each other and/or other resources (e.g., smart devices and theInternet) via a wireless router 101, depicted in room 252, and/or alocal mesh network. Additionally, other client devices -particularlymobile devices such as smart phones, tablets, laptops, wearable devices,etc.-may also be present, e.g., carried by one or more persons (e.g.,user 103) in the home and may or may not also be connected to the sameLAN. It should be understood that the configuration of client devicesdepicted in FIG. 2 is just one example; more or fewer and/or differentclient devices may be deployed across any number of other rooms and/orareas other than a home.

Further depicted in FIG. 2 are a plurality of smart devices. The smartdevices include a smart light 145 _(A1) that is provided by a first 3P(as indicated by the “A” designation). The smart light 145 _(A1) iscontrollable via a Bluetooth radio using a protocol suite, of the first3P, that is a variation of an industry standard. The smart devicesfurther include smart lights 145 _(B1) and 145 _(B2), and a smartdoorbell 145 _(B3), that are controlled via a hub 147 _(B) that iscoupled to the wireless router 101. The smart lights 145 _(B1) and 145_(B2), smart doorbell 145 _(B3), and hub 147 _(B) are provided by asecond 3P (as indicated by the “B” designation). The smart lights 145_(B1) and 145 _(B2), smart doorbell 145 _(B3), and hub 147 _(B) utilizean industry standard protocol, and the hub 147 _(B) is addressable viaTCP/IP, utilizing a Wi-Fi radio and via its hardwired connection towireless router 101. The smart devices further include smart washer 145_(C1) that is provided by a third 3P (as indicated by the “C”designation). The smart washer 145 _(C1) is controllable via a low-ratewireless personal area network radio channel using a proprietaryprotocol suite of the third 3P. The smart devices further include smartthermostat 145 _(D1) that is provided by a fourth 3P (as indicated bythe “D” designation). The smart thermostat 145 _(D1) is controllable viaa Wi-Fi radio using a variation, of the fourth 3P, of the UDP protocolsuite. It should be understood that the configuration of smart devices145 depicted in FIG. is just one example; more or fewer and/or differentsmart devices may be deployed across any number of other rooms and/orareas other than a home.

FIG. 2 and the above description of FIG. 2 will now be utilized indescribing various aspects of FIGS. 3-7 . FIG. 3 illustrates an examplestate diagram 300 of an example of controlling smart lights 145 _(B1)and 145 _(B2) of FIG. 2 , using client device 110 ₁ of FIG. 2 . In FIG.3 , adapter interaction engine 114 ₁ of client device 110 ₁ identifies ageneric smart device control command at 352. The generic smart devicecontrol command can be generated by client device 110 ₁, or provided toclient device 110 ₁ by remote cloud-based automated assistantcomponent(s) 120 and/or by another client device. The generic smartdevice control command identifies smart lights 145 _(B1) and 145 _(B2),indicates that their color should be adjusted to “bright red”, andoptionally identifies the second 3P.

At 354, the adapter interaction engine 114 ₁ selects 3P adapter 113 _(B)that corresponds to the second 3P. The adapter interaction engine 114 ₁can select the 3P adapter113_(B), from a plurality of available 3Padapters stored locally at the client device 110 ₁, based on the 3Padapter 113 _(B) being assigned to the second 3P and/or to the smartlights 145 _(B1) and 145 _(B2) specified in the generic smart devicecontrol command.

At 356, the interaction engine 114 ₁ provides the generic smart devicecontrol command to the 3P adapter 113 _(B). In some implementations, the3P adapter 113 _(B) is locally stored, and optionally already executing,at the client device 110 ₁ based on the smart lights 145 _(B1) and 145_(B2) and/or other smart device(s) of the second 3P being registered foran account associated with the client device 110 ₁. In someimplementations, the 3P adapter 113 _(B) is already executing at theclient device 110 ₁ further based on, for example, the 3P adapter 113_(B) being recently utilized at the client device110₁, being utilized inthe past at dates/times that are similar to a current date/time, and/orbased on one or more other criteria.

At 362, the 3P adapter 113 _(B) generates a specific command based onthe provided generic command and, at 364, the 3P adapter 113 _(B)returns the generated specific command to the adapter interaction engine114 ₁. It is noted that, even though the second 3P utilizes a standardprotocol, the generated specific command may differ from that whichwould otherwise be generated utilizing the standard protocol since, forexample, the 3P adapter 113 _(B) may integrate custom logic, of thesecond 3P, that will result in specific commands, for the “bright red”state change request, that are specifically tailored to the second 3P.The specific command can be addressed to the smart hub 147 _(B) that candirectly forward the specific command to the smart lights 145 _(B1) and145 _(B2), which can directly interpret the specific command to cause achange of the color state of the lights to the “bright red” state.Alternatively, the smart hub 147 _(B) can interpret the specific commandto thereby directly control the smart lights 145 _(B1) and 145 _(B2) inaccordance with the specific command.

At 358, the adapter interaction engine 114 ₁ selects a radio 112 _(B)for the 3P adapter 113 _(B) and provides the specific command to causeit to be transmitted using the selected radio at 372. The radio 112 _(B)is selected, from a plurality of available radios at the client device110 ₁, based on being assigned (locally at the client device 110 ₁) tothe 3P adapter 113 _(B). For example, the radio 112 _(B) can be a Wi-Firadio and the specific command can be transmitted to the smart hub 147_(B) via TCP/IP.

As noted above, the smart hub 147 _(B) can directly forward the specificcommand to the smart lights 145 _(B1) and 145 _(B2), which can directlyinterpret the specific command to cause a change of the color state ofthe lights to the “bright red” state. Alternatively, the smart hub 147_(B) can interpret the specific command and send further interpretedcommands to thereby directly control the smart lights 145 _(B1) and 145_(B2) in accordance with the specific command. Regardless, at 382 thestate change is implemented. Optionally, at 384 the smart hub 147 _(B)transmits a response back to the radio 112 _(B) and the response isprovided to the 3P adapter 113 _(B), which interprets the response(e.g., into a schema of the automated assistant) and provides theinterpreted response to the adapter interaction engine 114 ₁. Theadapter interaction engine 114 ₁ can optionally provide the interpretedresponse to one or more additional components, as a response to theidentified generic command. For example, where the interpreted responseindicates the state change was successful it can be provided to update adisplay of client device 110 ₁ to indicate the success, can be provideto cloud-based automated assistant component(s) 120 to update the homegraph 152 to reflect the state change, provided to a system 140associated with the second 3P to reflect the state change, and/orprovided to other component(s).

FIG. 4 illustrates an example state diagram 400 of an example of routinga generic smart device control command, for controlling smart light 145_(A1) of FIG. 2 , from client device 110 ₁ of FIG. 2 to client device110 ₂ of FIG. 2 . In FIG. 4 , routing engine 115 ₁ of client device 110₁ identifies a generic smart device control command at 452. The genericsmart device control command can be generated by client device 110 ₁, orprovided to client device 110 ₁ by remote cloud-based automatedassistant component(s) 120 and/or by another client device. For example,the generic smart device control command can be generated by clientdevice 110 ₁ responsive to touch-interaction with a graphical interfaceelement, displayed via a touchscreen of the client device 110 ₁, thatenables adjusting a dimming level of the smart light 145 _(A1). Thegeneric smart device control command identifies smart light 145 _(A1),indicates that it should be adjusted to a “50% dimmed” state, andoptionally identifies the first 3P.

At 454, the routing engine 115 ₁ determines that client device 110 ₁ hasan unreliable communication channel with the smart light 145 _(A1). Forexample, as mentioned above the smart light 145 _(A1) can becontrollable via a Bluetooth radio. Due to the distance between theclient device 110 ₁ and the smart light 145 _(A1), and/or due toobstructions, the Bluetooth radio of the client device 110 ₁ may nothave any communication with the smart light 145 _(A1) or may havecommunication, but with a signal strength that fails to satisfy athreshold. The routing engine 115 ₁ can determine the unreliablecommunication channel based on current data from the Bluetooth radio, orbased on past historical data (e.g., stored in a local mapping of clientdevices to smart devices).

At 456, the routing engine 115 ₁ resolves assistant client device 110 ₂that has a reliable communication channel with the smart light 145_(A1). In resolving the client device 110 ₂, the routing engine 115 ₁can optionally rely on a client devices to smart devices mapping, storedlocally at the client device 110 ₁, to determine client device 110 ₂ isone having an established reliable communications channel to the smartlight 145 _(A1) (e.g., as a result of being proximate to the smart light145 _(A1)). The client devices to smart devices mapping can defineclient devices of the ecosystem and, for each of the assistant clientdevices, can define corresponding smart device(s) for which theassistant client device has a corresponding reliable communicationschannel, and can optionally define signal strength(s) between theassistant client device and smart device(s). Such mapping can optionallybe included in a locally stored home graph as described herein.

At 458, the routing engine 115 ₁ transmits, using a local network, thegeneric smart device control command to the resolved client device 110₂.

At 460, the adapter interaction engine 114 ₂ of the client device 110 ₂selects 3P adapter 113 _(A) that corresponds to the first 3P. Theadapter interaction engine 114 ₂ can select the 3P adapter 113 _(A),from a plurality of available 3P adapters stored locally at the clientdevice 110 ₂, based on the 3P adapter 113 _(A) corresponding to thefirst 3P and/or the smart light 145 _(A1) specified in the generic smartdevice control command.

At 462, the interaction engine 114 ₂ provides the generic smart devicecontrol command to the 3P adapter 113 _(A). In some implementations, the3P adapter 113 _(A) is locally stored, and optionally already executing,at the client device 110 ₂ based on the smart light 145 _(A1) beingregistered for an account associated with the client device 110 ₂. Insome implementations, the 3P adapter 113 _(A) is already executing atthe client device 110 ₁ further based on, for example, there being astrong strength of signal between the client device 110 ₂ and the smartlight 145 _(A1), 3P adapter 113 _(A) being recently utilized at theclient device110₁, and/or based on one or more other criteria.

At 472, the 3P adapter 113 _(A) generates a specific command based onthe provided generic command and, at 474, the 3P adapter 113 _(A)returns the generated specific command to the adapter interaction engine114 ₂. It is noted that the fist 3P utilizes a protocol suite that is avariation of an industry standard and, as a result, the generatedspecific command may not conform to the industry standard. The specificcommand can be addressed to the smart light 145 _(A1), which candirectly interpret the specific command to cause a change of the dimminglevel of the smart light 145 _(A1) to “50%”.

At 464, the adapter interaction engine 114 ₂ selects a radio 112 _(A)for the 3P adapter 113 _(A) and causes it to be transmitted using theselected radio at 482. The radio 112 _(A) is selected, from a pluralityof available radios at the client device 110 ₂, based on being assigned(locally at the client device 110 ₂) to the 3P adapter 113 _(A). Forexample, the radio 112 _(A) can be a Bluetooth radio.

At 492 the smart light 145 _(A1) directly interprets the specificcommand to cause a change of the dimming level of the smart light 145_(A1) to “50%”. Optionally the smart light 145 _(A1) transmits aresponse back to the radio 112 _(A) and the response is provided to the3P adapter 113 _(A), which interprets the response (e.g., into a schemaof the automated assistant) and provides the interpreted response to theadapter interaction engine 114 ₂. The adapter interaction engine 114 ₂can optionally provide the interpreted response to one or moreadditional components, as a response to the identified generic command.

Turning now to FIGS. 5-7 , various methods are described with referenceto operations of flow charts of those figures. For convenience, theoperations of flow charts described below are described with referenceto a system that performs the operations. The system can include one ormore components of one or more client devices (e.g., client devices thatexecute an automated assistant client) and/or one or more components ofa separate computing system (e.g., remote automated assistantserver(s)). Moreover, different systems can perform the operations ofdifferent flowcharts. Additionally, while the operations of theflowcharts are shown in a particular order, this is not meant to belimiting. One or more operations may be reordered, omitted, or added.

FIG. 5 is a flow chart illustrating an example method of 500 controllinga smart device, using a local assistant client device, according tovarious implementations disclosed herein. FIG. 5 can be implementedusing one or more processors of the local assistant client device.

At block 502, the system identifies a generic smart device controlcommand that specifies smart device(s) and state(s) of the smartdevice(s) to be altered.

At block 504, the system determines whether there is a reliablecommunications channel.

If, at block 504, the system determines there is not a reliablecommunications channel, the system proceeds to block 506.

At block 506, the system selects an alternate local assistant clientdevice with a reliable communications channel to the smart device(s).

At block 508, the system transmits the generic smart device controlcommand to the selected alternate local assistant client device forlocally controlling the smart device(s). The selected alternate localassistant client device, upon receiving the generic smart device controlcommand, can locally process the generic smart device control command togenerate a specific command, that corresponds to the generic smartdevice control command, and that is directly interpretable by the smartdevice to effectuate alteration of the state at the smart device. Theselected alternate local assistant client device can further transmitthe specific command over the established reliable communicationschannel to effectuate the alteration of the state at the smart device.

If, at an iteration of block 504, the system determines there is areliable communications channel, the system proceeds to block 510.

At block 510, the system selects a particular adapter from a pluralityof available adapters and based on the generic smart device controlcommand.

At block 512, the system processes the generic smart device controlcommand using the selected adapter to generate specific commands.

At block 514, the system transmits the specific command over thereliable communications channel to effectuate an alteration of thestate(s) of the smart device(s).

FIG. 6 is a flow chart illustrating another example 600 method ofcontrolling a smart device, using a local assistant client device,according to various implementations disclosed herein. FIG. 6 can beimplemented using one or more processors of the local assistant clientdevice.

At block 602, the system preemptively executes each of a plurality ofadapters, such as a plurality of 3P adapters and optionally a 1Padapter.

At block 604, the system identifies a generic smart device controlcommand that specifies smart device(s) and state(s) of the smartdevice(s) to be altered.

At block 606, the system selects, based on the generic smart devicecontrol command, a particular adapter from the plurality of executingadapters.

At block 608, the system processes the generic smart device controlcommand using the selected adapter to generate a specific command.

At block 610, the system selects, from a plurality of availablecommunication channels, a particular communication channel assigned tothe selected adapter.

At block 612, the system transmits the specific command using theselected particular communication channel to cause the state(s) to bealtered at the smart device(s).

FIG. 7 is a flow chart illustrating an example method 700 of registeringa smart device, using a local assistant client device, according tovarious implementations disclosed herein. FIG. 7 can be implementedusing one or more processors of the local assistant client device and/orone or more processors of remote automated assistant server(s).

At block 702, the system causes each assistant client device, of anecosystem of assistant client devices, to scan communication channel(s)for an unprovisioned smart device(s).

At block 704, the system determines whether any new unprovisioned smartdevices are detected by the scanning.

If, at an iteration of block 704, the system determines there aren’t anynew unprovisioned smart devices, the system proceeds to block 706 andthe method 700 ends.

If at an iteration of block 704, the system determines there is at leastone new unprovisioned smart device, the system proceeds to block 708.

At block 708, the system causes a selected assistant device to interfacewith the new smart device, via a communications channel, to establishlocal connection(s) of the new smart device to the assistant clientdevice(s). Block 708 may optionally include one or more sub-blocks, suchas sub-block 709.

At sub-block 709, the system can select the assistant client device forinterfacing based on human interaction data for the assistant clientdevice. For example, at sub-block 709 the new smart device can be thesmart thermostat 145 _(D1) of FIG. 2 and it can be broadcasting anaccess point, but not yet connected to the secured wireless router 101utilized by the assistant client devices, and can lack the credentialsfor connecting to the secured wireless router 101. In some of thoseimplementations, each of a plurality of assistant client devices, of theecosystem of client devices, detected the availability of the accesspoint through scanning of its respective Wi-Fi radio. For example, allthree of the client devices 110 ₁₋₃ of FIG. 2 may have detected theavailability of the access point. In some versions of thoseimplementations, a single assistant client device is selected forconnecting to the smart device via the access point of the smart device,and securely sharing, with the smart device via the connection, thecredentials for connecting to the secured Wi-Fi network. Once thecredentials are shared, the smart device connects to the secured Wi-Finetwork and can be registered through communications via the Wi-Finetwork. The single assistant client device can be selected based atleast in part on human interaction data for the single assistant clientdevice. For example, client device 110 ₃ can be selected based on thehuman interaction data indicating that no user interface input has beenreceived at client device 110 ₃ within a threshold amount of time and/orbased on the human interaction data indicating that the client device110 ₃ is utilized less frequently than client device 110 ₁ and clientdevice 110 ₂.

At block 710, the system causes the selected or an additional assistantclient device to use an adapter, for the new smart device(s), ingenerating registration data for the new smart device. Block 710 mayoptionally include one or more sub-blocks. At sub-block 711, the systemcan provide the adapter to the assistant client device(s) in response tothe assistant client device(s) detecting the new smart device. Forexample, the adapter may not yet be locally stored at the assistantclient device(s) if other smart device(s), for the party associated withthe new smart device(s), are not yet linked with the assistant clientdevices. In such a situation, the adapter for that party can beretrieved from an adapter database, and provided to the assistant clientdevice(s).

At block 712, the system uses the registration data to incorporate thenew smart device into a home graph. Block 712 may optionally include oneor more sub-blocks. At sub-block 713, the system may store dataindicating the assistant client device(s) that can locally control thenew smart device, and optionally indicating preferences for locallycontrolling the new smart device. At sub-block 714, the system mayoptionally select the home graph based on biometric data, detected viaone or more of the assistant client devices, corresponding to an accountassociated with the home graph.

At block 715, the system may optionally transmit registrationinformation to a third-party for the new smart device.

FIG. 8 is a block diagram of an example computing device 810 that mayoptionally be utilized to perform one or more aspects of techniquesdescribed herein. Computing device 810 typically includes at least oneprocessor 814 which communicates with a number of peripheral devices viabus subsystem 812. These peripheral devices may include a storagesubsystem 824, including, for example, a memory subsystem 825 and a filestorage subsystem 826, user interface output devices 820, user interfaceinput devices 822, and a network interface subsystem 816. The input andoutput devices allow user interaction with computing device 810. Networkinterface subsystem 816 provides an interface to outside networks and iscoupled to corresponding interface devices in other computing devices.

User interface input devices 822 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touchscreen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and/or othertypes of input devices. In general, use of the term “input device” isintended to include all possible types of devices and ways to inputinformation into computing device 810 or onto a communication network.

User interface output devices 820 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem may also provide non-visual display such as via audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computing device 810 to the user or to another machine or computingdevice.

Storage subsystem 824 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein. For example, the storage subsystem 824 may include the logic toperform selected aspects of one or more methods described herein.

These software modules are generally executed by processor 814 alone orin combination with other processors. Memory 825 used in the storagesubsystem 824 can include a number of memories including a main randomaccess memory (RAM) 830 for storage of instructions and data duringprogram execution and a read only memory (ROM) 832 in which fixedinstructions are stored. A file storage subsystem 826 can providepersistent storage for program and data files, and may include a harddisk drive, a floppy disk drive along with associated removable media, aCD-ROM drive, an optical drive, or removable media cartridges. Themodules implementing the functionality of certain implementations may bestored by file storage subsystem 826 in the storage subsystem 824, or inother machines accessible by the processor(s) 814.

Bus subsystem 812 provides a mechanism for letting the variouscomponents and subsystems of computing device 810 communicate with eachother as intended. Although bus subsystem 812 is shown schematically asa single bus, alternative implementations of the bus subsystem may usemultiple busses.

Computing device 810 can be of varying types including a workstation,server, computing cluster, blade server, server farm, or any other dataprocessing system or computing device. Due to the ever-changing natureof computers and networks, the description of computing device 810depicted in FIG. 8 is intended only as a specific example for purposesof illustrating some implementations. Many other configurations ofcomputing device 810 are possible having more or fewer components thanthe computing device depicted in FIG. 8 .

In situations in which certain implementations discussed herein maycollect or use personal information about users (e.g., user dataextracted from other electronic communications, information about auser’s social network, a user’s location, a user’s time, a user’sbiometric information, and a user’s activities and demographicinformation, relationships between users, etc.), users are provided withone or more opportunities to control whether information is collected,whether the personal information is stored, whether the personalinformation is used, and how the information is collected about theuser, stored and used. That is, the systems and methods discussed hereincollect, store and/or use user personal information only upon receivingexplicit authorization from the relevant users to do so.

For example, a user is provided with control over whether programs orfeatures collect user information about that particular user or otherusers relevant to the program or feature. Each user for which personalinformation is to be collected is presented with one or more options toallow control over the information collection relevant to that user, toprovide permission or authorization as to whether the information iscollected and as to which portions of the information are to becollected. For example, users can be provided with one or more suchcontrol options over a communication network. In addition, certain datamay be treated in one or more ways before it is stored or used so thatpersonally identifiable information is removed. As one example, a user’sidentity may be treated so that no personally identifiable informationcan be determined. As another example, a user’s geographic location maybe generalized to a larger region so that the user’s particular locationcannot be determined.

In some implementations, a method implemented by one or more processorsof a client device executing an automated assistant client is provided,and includes identifying a generic smart device control command. Thegeneric smart device control command specifies at least one smart deviceand at least one state to be altered at the smart device. The methodfurther includes determining whether a reliable communications channelis established between the client device and the smart device and,responsive to determining the reliable communications channel is notestablished between the client device and the smart device: accessing aclient devices to smart devices mapping, stored locally at the clientdevice, to resolve an additional client device with an establishedreliable communications channel to the smart device; and transmitting,over a local network, the generic smart device control command to theadditional client device. Transmitting the generic smart device controlcommand to the additional client device causes the additional clientdevice to: locally process the generic smart device control command togenerate a specific command, that corresponds to the generic smartdevice control command, and that is directly interpretable by the smartdevice to effectuate alteration of the state at the smart device, andtransmit the specific command over the established reliablecommunications channel to effectuate the alteration of the state at thesmart device.

These and other implementations of the technology can optionally includeone or more of the following features.

In some implementations, the method further includes, responsive todetermining the reliable communications channel is established betweenthe client device and the smart device: selecting a particularthird-party (3P) adapter from a plurality of 3P adapters locallyexecutable at the client device, the selecting based on the smart devicecontrol command; processing the smart device control command using theselected 3P adapter to generate the specific command, that correspondsto the generic smart device control command, and that is directlyinterpretable by the smart device to effectuate alteration of the stateat the smart device; and transmitting, over the reliable communicationschannel established between the client device and the smart device, thespecific command to effectuate the alteration of the state at the smartdevice. In some versions of those implementations, the method furtherincludes, prior to the identifying the generic smart device controlcommand: receiving the particular 3P adapter at the client device andfrom a remote server, where receiving the particular 3P adapter isresponsive to the smart device, or an additional smart device of the 3P,being registered for the client device. In some additional oralternative versions of those implementations, the method furtherincludes, prior to the identifying the generic smart device controlcommand: preemptively executing, locally at the client device, theparticular 3P adapter to reduce latency in processing the smart devicecontrol command using the selected 3P adapter to generate the specificcommand.

In some implementations, determining whether the reliable communicationschannel is established between the client device and the smart deviceincludes determining whether any communications channel is establishedbetween the client device and the smart device and/or determiningwhether a communications channel, established between the client deviceand the smart device, has at least a threshold signal strength.Determining whether any communications channel is established anddetermining whether the communications channel has at least a thresholdsignal strength can be based on the client devices to smart devicesmapping and/or can be based on one or more signals received via thecommunications channel subsequent to a most recent updating of theclient devices to smart devices mapping.

In some implementations, accessing the client devices to smart devicesmapping, stored locally at the client device, to resolve the additionalclient device with the established reliable communications channel tothe smart device includes: determining, based on the client devices tosmart devices mapping, that a given signal strength, between theadditional client device and the smart device, is greatest amongst allclient devices of the client devices to smart devices mapping.

In some implementations, the established reliable communications channelto the smart device is a Bluetooth radio channel.

In some implementations, the generic smart device control command isgenerated locally at the client device by locally processing userinterface input received at the client device. In some of thoseimplementations, the user interface input is interaction, at atouchscreen of the client device, with a rendered graphical userinterface element corresponding to the smart device.

In some implementations, the user interface input is a voice inputreceived via at least one microphone of the client device, and locallyprocessing the user interface input includes using a speech-to-textprocessor to convert the voice input into text, and performing naturallanguage understanding on the text to generate the generic smart devicecontrol command.

In some implementations, the method further includes receiving voiceinput via at least one microphone of the client device and streaming thevoice input to a remote automated assistant system. In some versions ofthose implementations, identifying the generic smart device controlcommand includes receiving the generic smart device control command fromthe remote automated assistant client responsive to streaming the voiceinput to the remote automated assistant system.

In some implementations, a method implemented by one or more processorsof a client device executing an automated assistant client is providedand includes preemptively executing, locally at the client device, eachof a plurality of third-party (3P) adapters each generated by acorresponding 3P. Each of the 3P adapters converts each of a pluralityof corresponding generic smart device control commands intocorresponding specific commands that are each tailored, when locallytransmitted to at least one corresponding smart device of the 3P, to bedirectly interpretable by the corresponding smart device to effectuate astate change at the corresponding smart device, or at a correspondingadditional smart device directly controlled by the corresponding smartdevice. The plurality of 3P adapters are optionally preemptivelyexecuted locally at the client device responsive to smart devices of the3P being previously registered for the client device. The method furtherincludes identifying a particular generic smart device control commandthat specifies at least one particular smart device and at least onestate to be altered at the particular smart device. The method furtherincludes: selecting, from the plurality of 3P adapters and based on theparticular smart device control command, a particular 3P adapter;processing the particular smart device control command using theselected particular 3P adapter to generate a particular specificcommand, of the specific commands, that corresponds to the smart devicecontrol command; selecting, from a plurality of communication channelsavailable at the automated assistant client device, a particularcommunication channel assigned to the particular 3P adapter; andtransmitting, using the selected particular communication channel, theparticular specific command to cause the at least one state to bealtered at the particular smart device.

These and other implementations of the technology can optionally includeone or more of the following features.

In some implementations, the plurality of 3P adapters preemptivelyexecuting at the client device are a subset of available 3P adaptersstored at the client device. In some of those implementations, themethod further includes selecting the particular 3P adapter, from theavailable 3P adapters, for preemptively executing at the client device.The selecting can be based on one or more dynamic criteria, such as acorresponding recency of utilization of the particular 3P adapter at theclient device; a signal strength for a communication channel between theclient device and the particular smart device corresponding to theparticular 3P adapter; a time of day; and/or a day of the week.

In some implementations, the method further includes, prior to thepreemptively executing: receiving the particular 3P adapter at theclient device and from a remote server, optionally responsive to theparticular smart device, or an additional smart device of the 3P, beingregistered for the client device.

In some implementations, an additional 3P adapter, of the 3P adapterspreemptively executed at the client device, is assigned to an additionalparticular communication channel that is distinct from the particularcommunication channel. In some of those implementations, the particularcommunication channel is a first radio channel of the client device andthe additional particular communication channel is a second radiochannel of the client device. For example, the first radio channel canbe a Bluetooth channel and the second radio channel can be a Wi-Fi radiochannel.

In some implementations, the particular specific command generated bythe 3P adapter conforms to a particular protocol suite of the 3Padapter, and an additional 3P adapter, of the 3P adapters preemptivelyexecuted at the client device, generates commands that conform to anadditional communication protocol suite that is distinct from theparticular communication protocol suite. In some of thoseimplementations the additional 3P adapter and the 3P adapter are bothassigned to the particular communication channel. In some versions ofthose implementations, the particular protocol suite of the 3P adapterdoes not conform to an industry adopted standard, and can optionally beproprietary to the 3P.

In some implementations, the particular generic smart device controlcommand is generated responsive to user interface input received at anadditional client device, and identifying the particular generic smartdevice control command includes: receiving, via the local networkcommunication, the particular generic smart device command from theadditional client device. In some of those implementations, the methodfurther includes, prior to identifying the particular generic smartdevice control command: determining a signal strength, for theparticular communication channel, between the client device and theparticular smart device; and transmitting data indicative of the signalstrength. The particular generic smart device command can be receivedfrom the additional client device based on the transmitted dataindicative of the signal strength.

In some implementations, selecting, from the plurality of 3P adaptersand based on the particular smart device control command, the particular3P adapter includes selecting the 3P adapter based on: the generic smartdevice control command further specifying an identifier for the 3Padapter, and/or the particular smart device specified by the genericsmart device control command being assigned, at the client device, tothe 3P adapter.

In some implementations, the method further includes: identifying, basedon the generic smart device control command, an address for theparticular smart device; and verifying that the particular specificcommand is addressed to only the address for the particular smartdevice. Transmitting the particular smart device control command usingthe selected particular communication channel can be contingent on theverifying.

In some implementations, a method implemented by one or more processorsis provided and includes causing each assistant client device, of anecosystem of assistant client devices connected to a secured Wi-Finetwork that provides Internet access, to scan one or morecommunications channels for availability of an access point beingbroadcast by a smart device that lacks credentials for connecting to thesecured Wi-Fi network. The method further includes determining, based ondata generated by the scanning, that each of a plurality of assistantclient devices, of the ecosystem of client devices, detected theavailability of the access point. The method further includes selectinga single assistant client device from the plurality of assistant clientdevices that detected the availability of the access point. Selectingthe single assistant client device from the plurality of assistantclient devices can optionally be based at least in part on humaninteraction data for the single assistant client device. The methodfurther includes causing the single assistant client device to: break aconnection with the secured Wi-Fi network; after breaking theconnection, establish a connection to the smart device via the accesspoint, and securely share, with the smart device via the connection, thecredentials for connecting to the secured Wi-Fi network.

These and other implementations of the technology disclosed herein caninclude one or more of the following features.

In some implementations, selecting the single assistant client devicecan be based at least in part on the human interaction data for thesingle assistant client device and can include: selecting the singleassistant client device based on the human interaction data indicatingthat no user interface input has been received at the single assistantclient device within a threshold amount of time; selecting the singleassistant client device based on the human interaction data indicatingthat no audible output is being rendered by the assistant client deviceresponsive to previously received user interface input; and/or selectingthe single assistant client device based on the human interaction dataindicating that the single assistant client device is utilized lessfrequently than any other of the plurality of assistant client devicesthat detected the setup access point.

In some implementations, the method further includes: receiving, via thesecured Wi-Fi network, registration data from the smart device; andusing the registration data to assign the smart device in a stored homegraph associated with the ecosystem of assistant client devices. In someof those implementations, using the registration data to assign thesmart device in the stored home graph includes: assigning the smartdevice in the stored home graph based on determining that biometricdata, detected via one or more of the assistant client devices of theecosystem, corresponds to an account associated with the stored homegraph. For example, the stored home graph can be selected, based on thebiometric data, in lieu of an additional home graph associated with theecosystem of assistant client devices and associated with an additionalaccount. In various implementations the biometric data includes voicedata detected via one or more microphones of the one or more of theassistant client devices of the ecosystem.

In some implementations, a method implemented by one or more processorsis provided and includes receiving, from each of a plurality ofassistant client devices each having a corresponding short-range radio,a corresponding signal strength for a corresponding connection, via theshort-range radio, to a smart device. The method further includesdetermining, based on the received signal strengths, that a givenassistant device, of the plurality of assistant devices, is preferredfor locally controlling the smart device. The method further includesidentifying a generic smart device control command that specifies atleast one smart device and at least one state to be altered at the smartdevice. The method further includes, based on determining the givenassistant device is preferred for locally controlling the smart device,transmitting the generic smart device control command to the givenassistant device to cause the given assistant device to: locally processthe generic smart device control command to generate a specific command,that corresponds to the generic smart device control command, and thatis directly interpretable by the smart device to effectuate alterationof the state at the smart device and transmit the specific command overthe short-range radio to effectuate the alteration of the state at thesmart device.

In some implementations, a method implemented by one or more processorsis provided and includes causing each assistant client device, of anecosystem of assistant client devices, to scan one or morecommunications channels for any unprovisioned smart devices. The methodfurther includes determining, based on data generated by the scanning,that at least a given assistant client device, of the ecosystem ofclient devices, detected a given smart device that is unprovisioned. Themethod further includes causing the given assistant client device toutilize a locally stored third-party (3P) adapter to processregistration data, provided by the given smart device in a format forthe 3P, to generate registration information that is in a schema for anautomated assistant associated with the assistant client device. Themethod further includes utilizing the registration information toregister the given assistant client device in association with anaccount associated with the ecosystem of assistant client devices.

In some implementations, a method implemented by one or more processorsis provided and includes receiving, from each of a plurality ofassistant client devices each having a corresponding short-range radio,a corresponding signal strength for a corresponding connection, via theshort-range radio, to a smart device. The method further includesdetermining, based on the received signal strengths, that a givenassistant device, of the plurality of assistant devices, is preferredfor locally controlling the smart device. The method further includes,based on determining the given assistant device is preferred for locallycontrolling the smart device, causing a particular third-party (3P)adapter, that corresponds to the smart device, to be downloaded to, andlocally stored at, the given assistant device. The particular 3P adaptercan be, for example, configured to locally process generic smart devicecontrol commands to generate specific commands that correspond to thegeneric smart device control commands and that are directlyinterpretable by the smart device to effectuate alteration of the stateat the smart device.

These and other implementations of the technology disclosed herein caninclude one or more of the following features.

The method can further include, based on determining the given assistantdevice is preferred for locally controlling the smart device, causingthe particular 3P adapter to be preemptively executed at the givenassistant device.

In some implementations, the particular 3P adapter is downloaded to onlythe given assistant device, to the exclusion of other of the smartdevices, based on determining that the given assistant device ispreferred for locally controlling the smart device.

What is claimed is:
 1. A method implemented by one or more processors,the method comprising: causing each assistant client device, of anecosystem of assistant client devices connected to a secured Wi-Finetwork that provides Internet access, to scan one or morecommunications channels for availability of an access point beingbroadcast by a smart device that lacks credentials for connecting to thesecured Wi-Fi network; determining, based on data generated by thescanning, that each of a plurality of assistant client devices, of theecosystem of client devices, detected the availability of the accesspoint; selecting a single assistant client device from the plurality ofassistant client devices that detected the availability of the accesspoint, wherein selecting the single assistant client device from theplurality of assistant client devices is based at least in part on humaninteraction data for the single assistant client device; and causing thesingle assistant client device to: break a connection with the securedWi-Fi network, after breaking the connection, establish a connection tothe smart device via the access point, and securely share, with thesmart device via the connection, the credentials for connecting to thesecured Wi-Fi network.
 2. The method of claim 1, wherein selecting thesingle assistant client device based at least in part on the humaninteraction data for the single assistant client device comprises:selecting the single assistant client device based on the humaninteraction data indicating that no user interface input has beenreceived at the single assistant client device within a threshold amountof time.
 3. The method of claim 1, wherein selecting the singleassistant client device based at least in part on the human interactiondata for the single assistant client device comprises: selecting thesingle assistant client device based on the human interaction dataindicating that no audible output is being rendered by the assistantclient device responsive to previously received user interface input. 4.The method of claim 1, wherein selecting the single assistant clientdevice based at least in part on the human interaction data for thesingle assistant client device comprises: selecting the single assistantclient device based on the human interaction data indicating that thesingle assistant client device is utilized less frequently than anyother of the plurality of assistant client devices that detected thesetup access point.
 5. The method of claim 1, further comprising:receiving, via the secured Wi-Fi network, registration data from thesmart device; and using the registration data to assign the smart devicein a stored home graph associated with the ecosystem of assistant clientdevices.
 6. The method of claim 5, wherein using the registration datato assign the smart device in the stored home graph comprises: assigningthe smart device in the stored home graph based on determining thatbiometric data, detected via one or more of the assistant client devicesof the ecosystem, corresponds to an account associated with the storedhome graph.
 7. The method of claim 6, wherein assigning the smart devicein the stored home graph based on determining that the biometric datacorresponds to the account associated with the stored home graphcomprises: selecting, based on the biometric data, the stored home graphin lieu of an additional home graph associated with the ecosystem ofassistant client devices and associated with an additional account. 8.The method of claim 6, wherein the biometric data includes voice datadetected via one or more microphones of the one or more of the assistantclient devices of the ecosystem.
 9. The method of claim 5, furthercomprising: causing the single assistant client device to utilize alocally stored third-party (3P) adapter to process the registrationdata, provided by the smart device in a format for the 3P, to generateregistration information in a schema for an automated assistantassociated with the assistant client device.
 10. The method of claim 9,wherein the 3P adapter is specifically tailored by the 3P.
 11. Themethod of claim 10, wherein the 3P adapter integrates custom logic ofthe 3P.
 12. A system comprising: memory storing instructions; and one ormore processors operable to execute the instructions, stored in thememory, to perform the operations of: causing each assistant clientdevice, of an ecosystem of assistant client devices connected to asecured Wi-Fi network that provides Internet access, to scan one or morecommunications channels for availability of an access point beingbroadcast by a smart device that lacks credentials for connecting to thesecured Wi-Fi network; determining, based on data generated by thescanning, that each of a plurality of assistant client devices, of theecosystem of client devices, detected the availability of the accesspoint; selecting a single assistant client device from the plurality ofassistant client devices that detected the availability of the accesspoint, wherein selecting the single assistant client device from theplurality of assistant client devices is based at least in part on humaninteraction data for the single assistant client device; and causing thesingle assistant client device to: break a connection with the securedWi-Fi network, after breaking the connection, establish a connection tothe smart device via the access point, and securely share, with thesmart device via the connection, the credentials for connecting to thesecured Wi-Fi network.
 13. The system of claim 12, wherein selecting thesingle assistant client device based at least in part on the humaninteraction data for the single assistant client device comprises:selecting the single assistant client device based on the humaninteraction data indicating that no user interface input has beenreceived at the single assistant client device within a threshold amountof time.
 14. The system of claim 12, wherein selecting the singleassistant client device based at least in part on the human interactiondata for the single assistant client device comprises: selecting thesingle assistant client device based on the human interaction dataindicating that no audible output is being rendered by the assistantclient device responsive to previously received user interface input.15. The system of claim 12, wherein selecting the single assistantclient device based at least in part on the human interaction data forthe single assistant client device comprises: selecting the singleassistant client device based on the human interaction data indicatingthat the single assistant client device is utilized less frequently thanany other of the plurality of assistant client devices that detected thesetup access point.
 16. The system of claim 12, wherein the operationsfurther comprise: receiving, via the secured Wi-Fi network, registrationdata from the smart device; using the registration data to assign thesmart device in a stored home graph associated with the ecosystem ofassistant client devices.
 17. The system of claim 16, wherein using theregistration data to assign the smart device in the stored home graphcomprises: assigning the smart device in the stored home graph based ondetermining that biometric data, detected via one or more of theassistant client devices of the ecosystem, corresponds to an accountassociated with the stored home graph.
 18. The system of claim 17,wherein assigning the smart device in the stored home graph based ondetermining that the biometric data corresponds to the accountassociated with the stored home graph comprises: selecting, based on thebiometric data, the stored home graph in lieu of an additional homegraph associated with the ecosystem of assistant client devices andassociated with an additional account.
 19. The system of claim 17,wherein the biometric data includes voice data detected via one or moremicrophones of the one or more of the assistant client devices of theecosystem.
 20. The system of claim 16, wherein the operations furthercomprise: causing the single assistant client device to utilize alocally stored third-party (3P) adapter to process the registrationdata, provided by the smart device in a format for the 3P, to generateregistration information in a schema for an automated assistantassociated with the assistant client device.